Ascend PyTorch Profiler APIs

Restrictions

Ascend PyTorch Profiler APIs support multiple profiling methods, but these methods cannot be enabled at the same time.

Prerequisites

• Ensure that operations in Before You Start have been completed.
• Prepare a model trained on PyTorch 2.1.0 or later and a matched dataset, and port the model to the Ascend AI Processor. For details, see "Porting Adaptation" in the PyTorch Training Model Porting and Tuning Guide .

Profile Data Sampling and Parsing (torch_npu.profiler.profile)

1. Add the following sample code to the training script (for example, train_*.py)/online inference script to configure profile data sampling parameters, and then start the training/online inference. The following is a code example:
   

   For details about the torch_npu.profiler.profile API in the following sample code, see Ascend PyTorch Profiler APIs.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

   import torch import torch_npu ... experimental_config = torch_npu.profiler._ExperimentalConfig( export_type=torch_npu.profiler.ExportType.Text, profiler_level=torch_npu.profiler.ProfilerLevel.Level0, msprof_tx=False, aic_metrics=torch_npu.profiler.AiCMetrics.AiCoreNone, l2_cache=False, op_attr=False, data_simplification=False, record_op_args=False, gc_detect_threshold=None ) with torch_npu.profiler.profile( activities=[ torch_npu.profiler.ProfilerActivity.CPU, torch_npu.profiler.ProfilerActivity.NPU ], schedule=torch_npu.profiler.schedule(wait=0, warmup=0, active=1, repeat=1, skip_first=1), on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./result"), record_shapes=False, profile_memory=False, with_stack=False, with_modules=False, with_flops=False, experimental_config=experimental_config) as prof: for step in range(steps): train_one_step(step, steps, train_loader, model, optimizer, criterion) prof.step()

   Or

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

   import torch import torch_npu ... experimental_config = torch_npu.profiler._ExperimentalConfig( export_type=torch_npu.profiler.ExportType.Text, profiler_level=torch_npu.profiler.ProfilerLevel.Level0, msprof_tx=False, aic_metrics=torch_npu.profiler.AiCMetrics.AiCoreNone, l2_cache=False, op_attr=False, data_simplification=False, record_op_args=False, gc_detect_threshold=None ) prof = torch_npu.profiler.profile( activities=[ torch_npu.profiler.ProfilerActivity.CPU, torch_npu.profiler.ProfilerActivity.NPU ], schedule=torch_npu.profiler.schedule(wait=0, warmup=0, active=1, repeat=1, skip_first=1), on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./result"), record_shapes=False, profile_memory=False, with_stack=False, with_modules=False, with_flops=False, experimental_config=experimental_config) prof.start() for step in range(steps): train_one_step() prof.step() prof.stop()

   In addition to using tensorboard_trace_handler to export profile data, you can also use the following method:

   1 2 3 4 5 6 7 8 9

   import torch import torch_npu ... with torch_npu.profiler.profile() as prof: for step in range(steps): train_one_step(step, steps, train_loader, model, optimizer, criterion) prof.export_chrome_trace('./chrome_trace_14.json')

2. Parse profile data.

   Automatic parsing (see tensorboard_trace_handler and prof.export_chrome_trace in the preceding sample code) and offline parsing are supported.

3. View the sampled PyTorch training/online inference profile data result files and profile data analysis.

   For details about profile data result files, see Data Storing Directories.

Profile Data Sampling and Parsing (dynamic_profile)

dynamic_profile is used to start the profiling process at any time during model training/online inference.

dynamic_profile can be enabled in multiple methods. However, only one method can be enabled at a time.

Using environment variables

1. Configure the following environment variables:

   export PROF_CONFIG_PATH="profiler_config_path"

   After this environment variable is configured and training is started, dynamic_profile automatically creates the template file profiler_config.json in profiler_config_path. You can modify configuration options based on the template file.

   

   • This method applies only to training scenarios.
   • In this method, dynamic_profile cannot sample data of the first iteration (step 0).
   • This method depends on the profiling steps in the training process of the Torch native Optimizer.step(). Custom optimizer is not supported.
   • profiler_config_path can contain only letters, digits, and underscores (_). Soft links are not supported.
2. Start a training job.
3. Open a new CLI and modify the profiler_config.json configuration file to enable the profiling task.
   The configuration file contains the profile data collection parameters of Profiler. You can see profiler_config.json File Description to modify the parameters in the configuration file to execute different profiling tasks.

   

   • dynamic_profile determines whether the profiler_config.json file is modified by identifying the status of the file.
     • dynamic_profile polls every two seconds. If the profiler_config.json file is modified, the sampling process is started. Then, the running interval between step 10 and step 11 is recorded and used as the new polling interval. The minimum interval is one second.
     • If the profiler_config.json file is modified during the dynamic_profile sampling process, the dynamic_profile sampling is started again after the sampling process ends.
   • You are advised to use shared storage to set profiler_config_path of dynamic_profile.
   • The dynamic_profile maintenance and test logs are automatically recorded in the profiler_config_path directory. For details, see Using dynamic_profile for Dynamic Profiling of Maintenance and Test Logs.
4. Parse profile data.

   Automatic parsing and manual parsing are supported. For details, see the analyse parameter in Table 5.

5. View the sampled PyTorch training/online inference profile data result files and profile data analysis.

   For details about profile data result files, see Data Storing Directories.

Modifying the user training/online inference script by adding the dynamic_profile API

1. Add the following sample code to the training script (for example, train_*.py)/online inference script:

   1 2 3 4 5 6 7 8 9

   # Load the dynamic_profile module. from torch_npu.profiler import dynamic_profile as dp # Set the profiling configuration file path. dp.init("profiler_config_path") ... for step in steps: train_one_step() # Divide steps. dp.step()

   During init, dynamic_profile automatically creates the template file profiler_config.json in profiler_config_path. You can modify configuration items based on the template file.

   profiler_config_path can contain only letters, digits, and underscores (_). Soft links are not supported.

2. Start a training/online inference task.
3. Open a new CLI and modify the profiler_config.json configuration file to enable the profiling task.
   The configuration file contains the profile data collection parameters of Profiler. You can see profiler_config.json File Description to modify the parameters in the configuration file to execute different profiling tasks.

   

   • dynamic_profile determines whether the profiler_config.json file is modified by identifying the status of the file.
     • dynamic_profile polls every two seconds. If the profiler_config.json file is modified, the sampling process is started. Then, the running interval between step 10 and step 11 is recorded and used as the new polling interval. The minimum interval is one second.
     • If the profiler_config.json file is modified during the dynamic_profile sampling process, the dynamic_profile sampling is started again after the sampling process ends.
   • You are advised to use shared storage to set profiler_config_path of dynamic_profile.
   • The dynamic_profile maintenance and test logs are automatically recorded in the profiler_config_path directory. For details, see Using dynamic_profile for Dynamic Profiling of Maintenance and Test Logs.
4. Parse profile data.

   Automatic parsing and manual parsing are supported. For details, see the analyse parameter in Table 5.

5. View the sampled PyTorch training/online inference profile data result files and profile data analysis.

   For details about profile data result files, see Data Storing Directories.

Modifying the user training/online inference script by adding the dp.start() function of dynamic_profile

1. Add the following sample code to the training script (for example, train_*.py)/online inference script:

   1 2 3 4 5 6 7 8 9 10 11 12

   # Load the dynamic_profile module. from torch_npu.profiler import dynamic_profile as dp # Set the path of the profiling configuration file of the init API. dp.init("profiler_config_path") ... for step in steps: if step==5: # Set the path of the profiling configuration file of the start API. dp.start("start_config_path") train_one_step() # Divide steps. The code that requires profiling must be loaded between the dp.start() API and the dp.step() API. dp.step()

   start_config_path is also specified as the profiler_config.json path. However, you need to manually create a configuration file by referring to profiler_config.json File Description and set parameters based on your actual needs. The file name must be specified, for example, dp.start("/home/xx/start_config_path/profiler_config.json").

   profiler_config_path and start_config_path can contain only letters, digits, and underscores (_). Soft links are not supported.

   

   • After the dp.start() function is added, when a training or online inference task is proceeded to dp.start(), data is automatically sampled based on the profiler_config.json file specified by start_config_path. The dp.start() function does not detect the modification of the profiler_config.json file. It triggers a sampling task only during training or online inference.
   • After the dp.start() function is added and the training/online inference is started:
     • If the profiler_config.json configuration file is not specified in dp.start() or the configuration file does not take effect due to an error, sample data based on the profiler_config.json configuration file in the profiler_config_path directory when the task is proceeded to dp.start()
     • If the script proceeds to dp.start() when the dynamic_profile configured in dp.init() is valid, dp.start() does not take effect.
     • If the script proceeds to dp.start() after the dynamic_profile finishes sampling as specified in dp.init(), the script continues sampling with dp.start() and generates a new profile data file directory in the prof_dir directory.
     • If the profiler_config.json file in the profiler_config_path directory is modified while the dynamic_profile configured in dp.start() is valid, dp.init() starts after the dp.start() sampling finishes and a new profile data file is generated in the prof_dir directory.
   • You are advised to use shared storage to set profiler_config_path of dynamic_profile.
2. Start a training/online inference task.
3. Parse profile data.

   Automatic parsing and manual parsing are supported. For details, see the analyse parameter in Table 5.

4. View the sampled PyTorch training/online inference profile data result files and profile data analysis.

   For details about profile data result files, see Data Storing Directories.

Profile Data Sampling and Parsing (torch_npu.profiler._KinetoProfile)

1. Add the following sample code to the training script (for example, train_*.py)/online inference script to configure profile data sampling parameters, and then start the training/online inference. The following is a code example:
   

   For details about the torch_npu.profiler._KinetoProfile API in the following sample code, see Ascend PyTorch Profiler APIs.

   1 2 3 4 5 6 7 8 9 10 11 12 13

   import torch import torch_npu ... prof = torch_npu.profiler._KinetoProfile(activities=None, record_shapes=False, profile_memory=False, with_stack=False, with_flops=False, with_modules=False, experimental_config=None) for epoch in range(epochs): trian_model_step() if epoch == 0: prof.start() if epoch == 1: prof.stop() prof.export_chrome_trace("result_dir/trace.json")

   In this method, schedule and tensorboard_trace_handler cannot be used to export profile data.

2. Parse profile data.

   Automatic parsing is supported. For details, see prof.export_chrome_trace in the preceding sample code.

3. View the sampled PyTorch training/online inference profile data result files and profile data analysis.

   For details about profile data result files, see Data Storing Directories.

(Optional) Sampling and Parsing msprof_tx

In large cluster scenarios, traditional profiling involves a large amount of data and complex analysis process. You can use the msprof_tx parameter of experimental_config to enable the custom dotting function, customize the collection period or the start and end time of key functions, and identify key functions or iterations to quickly demarcate performance issues.

The usage and sample code are as follows:

1. Enable torch_npu.profiler and msprof_tx, and set profiler_level to Level_none (the level can be configured based on the actual collection requirements) to collect dotting data.
2. In the PyTorch script, call the mark APIs of torch_npu.npu.mstx, torch_npu.npu.mstx.mark, torch_npu.npu.mstx.range_start, torch_npu.npu.mstx.range_end and torch_npu.npu.mstx.mstx_range to collect the time required for sampling events. For details about the APIs, see "Ascend Extension for PyTorch Custom API" > "torch_npu.npu" > "profiler" in Ascend Extension for PyTorch API Reference .

Only the range duration on the host is recorded.

1
2
3

id = torch_npu.npu.mstx.range_start("dataloader", None) # If the second input parameter is set to None or not set, only the range duration on the host is recorded.
dataloader()
torch_npu.npu.mstx.range_end(id)

Mark IDs on compute streams to record the range time on the host and device.

1
2
3
4

stream = torch_npu.npu.current_stream()
id = torch_npu.npu.mstx.range_start("matmul", stream)    # Set the second input parameter to a valid stream, record the range time on the host and device.
torch.matmul()
torch_npu.npu.mstx.range_end(id)

Mark IDs on the collective communication stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from torch.distributed.distributed_c10d import _world

if (torch.__version__ != '1.11.0') :
 stream_id = _world.default_pg._get_backend(torch.device('npu'))._get_stream_id(False)
	collective_stream = torch.npu.Stream(stream_id=collective_stream_id, device_type=20, device_index=device_id)    # Use device_index to specify device IDs of actual services.
else:
 stream_id = _world.default_pg._get_stream_id(False)
 current_stream = torch.npu.current_stream()
 cdata = current_stream._cdata & 0xffff000000000000
	collective_stream = torch.npu.Stream(_cdata=( stream_id + cdata), device_index=device_id)    # Use device_index to specify device IDs of actual services.
id = torch_npu.npu.mstx.range_start("allreduce", collective_stream)    # Set the second input parameter to a valid stream, record the range time on the host and device.
torch.allreduce()
torch_npu.npu.mstx.range_end(id)

Mark IDs on the P2P communication stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from torch.distributed.distributed_c10d import _world

if (torch.__version__ != '1.11.0') :
 stream_id = _world.default_pg._get_backend(torch.device('npu'))._get_stream_id(True)
	p2p_stream = torch.npu.Stream(stream_id=collective_stream_id, device_type=20, device_index=device_id)    # Use device_index to specify device IDs of actual services.
else:
 stream_id = _world.default_pg._get_stream_id(True)
 current_stream = torch.npu.current_stream()
 cdata = current_stream._cdata & 0xffff000000000000
	p2p_stream = torch.npu.Stream(_cdata=( stream_id + cdata), device_index=device_id)    # Use device_index to specify device IDs of actual services.
id = torch_npu.npu.mstx.range_start("send", p2p_stream)    # Set the second input parameter to a valid stream, record the range time on the host and device.
torch.send()
torch_npu.npu.mstx.range_end(id)

To collect data in these scenarios, you need to configure the torch_npu.profiler.profile API and enable the msprof_tx switch. The following is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import torch_npu

stream = torch_npu.npu.current_stream()
id = torch_npu.npu.mstx.range_start("Func", stream)    # Mark the start of the range of the func function on the host and device.
func()            # Service code.
torch_npu.npu.mstx.range_end(id)            # Mark the end of the range of the func function on the host and device.

experimental_config = torch_npu.profiler._ExperimentalConfig(
 profiler_level=torch_npu.profiler.ProfilerLevel.Level_none,
 msprof_tx=True,
 export_type=torch_npu.profiler.ExportType.Db
 )
with torch_npu.profiler.profile(
 schedule=torch_npu.profiler.schedule(wait=1, warmup=1, active=2, repeat=2, skip_first=1),
 on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./result"),
 experimental_config=experimental_config) as prof:
       
 for epoch in range(epochs):
		torch_npu.npu.mstx.mark("train epoch start")    # Mark the instantaneous moment on the host and device. You can also use torch_npu.npu.mstx().mark("train epoch start").
  for step in range(steps):
   train_one_step(step, steps, train_loader, model, optimizer, criterion)
   prof.step()

By default, the msprof_tx function collects communication operator profile data in the format of comm:{communication name},{communicator name},{input data format},{input data counts}. For example, comm:HcclBroadcast,xxxxxx,int64,5, where xxxxxx indicates the communicator name.

This function allows you to view the execution and scheduling status of user-defined dotting from the framework to the CANN layer and then to the NPU, helping you identify key functions or events to be observed and demarcate performance issues.

For details about msprof_tx sampling results, see msproftx Data Description.

(Optional) Sampling Environment Variable Information

The Ascend PyTorch Profiler API profiles environment variable information by default. The following environment variables can be sampled:

• "ASCEND_GLOBAL_LOG_LEVEL"
• "HCCL_RDMA_TC"
• "HCCL_RDMA_SL"
• "ACLNN_CACHE_LIMIT"

Procedure:

1. Configure environment variables. The following is an example:

   export ASCEND_GLOBAL_LOG_LEVEL=1
   export HCCL_RDMA_TC=0
   export HCCL_RDMA_SL=0
   export ACLNN_CACHE_LIMIT=4096

   Set the environment variables based on the actual requirements.

2. Call the Ascend PyTorch Profiler API for sampling.
3. View the result data.
   • When the export_type of experimental_config is set to torch_npu.profiler.ExportType.Text, the environment variables configured in the preceding steps are stored in the profiler_metadata.json file in the {worker_name}_{timestamp}_ascend_pt directory.
   • When export_type of the experimental_config parameter is set to torch_npu.profiler.ExportType.Db, write environment variable information into the META_DATA table in the ascend_pytorch_profiler_{rank_id}.db file.

(Optional) Marking the Profile Data Collection Process With Custom Character String Keys and Values

• Example 1

  1 2	with torch_npu.profiler.profile(...) as prof: prof.add_metadata(key, value)

• Example 2

  1 2	with torch_npu.profiler._KinetoProfile(...) as prof: prof.add_metadata_json(key, value)

add_metadata and add_metadata_json can be configured under torch_npu.profiler.profile and torch_npu.profiler._KinetoProfile. They need to be added in the code of the profile data collection process, after profiler initialization and before finalization.

1

prof.add_metadata("test_key1", "test_value1")

1

prof.add_metadata_json("test_key2", [1, 2, 3])

The metadata passed by calling this API is written to the profiler_metadata.json file in the root directory of the collection result of the Ascend PyTorch Profiler API.

(Optional) Video Memory Visualization

The function classifies and displays the occupied data when the training process occupies the storage space during model training. Export the visualization file memory_timeline.html through export_memory_timeline. To output an HTML file, you need to install matplotlib in the Python environment and set torch_npu.profiler.profile to True. In addition, if you use this function, an ascend_pt data file is generated in the current directory. The following is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import torch
import torch_npu
...

def trace_handler(prof: torch_npu.profiler.profile):
 prof.export_memory_timeline(output_path="./memory_timeline.html", device="npu:0")

with torch_npu.profiler.profile(
 activities=[
  torch_npu.profiler.ProfilerActivity.CPU,
  torch_npu.profiler.ProfilerActivity.NPU
 ],
 schedule=torch_npu.profiler.schedule(wait=0, warmup=0, active=4, repeat=1, skip_first=0),
 on_trace_ready=trace_handler,
	record_shapes=True,           # Set it to True.
	profile_memory=True,          # Set it to True.
	with_stack=True,              # Set either with_stack or with_modules to True.
 with_modules=True
) as prof:
 for _ in range(steps):
  ...
  prof.step()

After data sampling, the memory_timeline.html file is exported, with the following visualization effect:

Figure 1 memory_timeline

• Time (ms): Horizontal coordinate, indicating the memory occupation time of the tensors (unit: ms).
• Memory (GB): Vertical coordinate, indicating the memory size occupied by the tensors (unit: GB).
• Max memory allocated: Allocated maximum memory size (unit: GB).
• Max memory reserved: Reserved maximum memory size (unit: GB).
• PARAMETER: Model parameters and model weights.
• OPTIMIZER_STATE: Optimizer status. For example, the Adam optimizer records some statuses during model training.
• INPUT: Input data.
• TEMPORARY: Temporarily occupied. It is defined as tensors that are allocated and then released for a single operator. Generally, these tensors store intermediate values.
• ACTIVATION: Activation values obtained in forward computation.
• GRADIENT: Gradient value.
• AUTOGRAD_DETAIL: Memory usage generated during backward computation.
• UNKNOWN: Unknown type.

Ascend PyTorch Profiler APIs

1
2

pid = os.getpid()
prof.export_chrome_trace(f'./chrome_trace_{pid}.json')

1

export_stacks('resule_dir/stack.log', metric='self_npu_time_total')

git clone https://github.com/brendangregg/FlameGraph
cd FlameGraph
./flamegraph.pl –title "NPU time" –countname "us." profiler.stacks > perf_viz.svg

1

export_memory_timeline(output_path="./memory_timeline.html", device="npu:0")

1
2
3
4
5
6
7
8

prof = torch_npu.profiler.profile(
on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./result"))
for step in range(steps):
    if step == 5:
        prof.start()
    train_one_step()
    if step == 5:
        prof.stop()

Profile data occupies certain disk space. As a result, the server may be unavailable when the disk space is used up. The space required by profile data is closely related to the model parameters, collection configurations, and number of collection iterations. You need to ensure that the available disk space in the directory where profile data is flushed is sufficient.

profiler_config.json File Description

The content of the profiler_config.json file is as follows (the default settings are used as an example):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

{
 "activities": ["CPU", "NPU"],
 "prof_dir": "./",
 "analyse": false,
 "record_shapes": false,
 "profile_memory": false,
 "with_stack": false,
 "with_flops": false,
 "with_modules": false,
 "active": 1,
 "start_step": 0,
 "is_rank": false,
 "rank_list": [],
 "experimental_config": {
  "profiler_level": "Level0",
  "aic_metrics": "AiCoreNone",
  "l2_cache": false,
  "op_attr": false,
  "gc_detect_threshold": null,
  "data_simplification": true,
  "record_op_args": false,
  "export_type": "text",
  "msprof_tx": false
 }
}

1
2
3
4
5
6
7

 "metadata": {
  "distributed_args":{
   "tp":2,
   "pp":4,
   "dp":8
  }
 }

experimental_config Parameter Description (dynamic_profile Scenario)

All experimental_config parameters are optional. The following table lists the sampling items that can be extended.

experimental_config Parameter Description

All experimental_config parameters are optional. The following table lists the sampling items that can be extended.

torch_npu.profiler.schedule Parameter Description

The torch_npu.profiler.schedule class parameters are used to set the sampling behavior in different steps in the sampling process. Prototype

torch_npu.profiler.schedule (wait, active, warmup = 0, repeat = 0, skip_first = 0)

The following figure shows the relationships between the torch_npu.profiler.schedule class, the step, and the on_trace_ready function.

Figure 2 Relationships between the torch_npu.profiler.schedule class, the step, and the on_trace_ready function

A code example of the configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

with torch_npu.profiler.profile(
    activities=[
        torch_npu.profiler.ProfilerActivity.CPU,
        torch_npu.profiler.ProfilerActivity.NPU,
    ],
    schedule=torch_npu.profiler.schedule(
        wait=1,                        # Waiting phase. One step is skipped.
        warmup=1,                      # Warm-up phase. One step is skipped.
        active=2,                      # Record the activity data of two steps and call on_trace_ready.
        repeat=2,                      # Repeat the wait+warmup+active process twice.
        skip_first=1                   # Skip one step.
    ),
    on_trace_ready=torch_npu.profiler.tensorboard_trace_handler('./result')
    ) as prof:
        for _ in range(9):
            train_one_step()
            prof.step()                #  Notify the profiler to finish a step.

Using dynamic_profile for Dynamic Profiling of Maintenance and Test Logs

1
2
3
4
5
6
7
8

profiler_config_path/
├── log
│    ├── dp_ubuntu_xxxxxx_rank_*.log
│    ├── dp_ubuntu_xxxxxx_rank_*.log.1
│    ├── monitor_dp_ubuntu_xxxxxx_rank_*.log
│    ├── monitor_dp_ubuntu_xxxxxx_rank_*.log.1
├── profiler_config.json
└── shm

• dp_ubuntu_xxxxxx.log: Execution log of dynamic_profile, which records all actions (INFO), warnings (WARNING), and errors (ERROR) during dynamic profiling. File naming format: dp_{Operating system}_{AI task process ID}_{rank_id}.log.

  When an AI task is started, each Rank will initiate an AI task process. The dynamic_profile generates log files for each AI task process based on the process ID of each task.

• dp_ubuntu_xxxxxx.log.1: This is a log aging backup file. The storage limit for the dp_ubuntu_xxxxxx.log file is 200 KB. Once the limit is reached, the earliest log entries are moved to dp_ubuntu_xxxxxx.log.1. The storage limit for the dp_ubuntu_xxxxxx.log.1 file is also 200 KB, and once the limit is reached, the earliest log entries are deleted through aging.
• monitor_dp_ubuntu_xxxxxx.log: This is the log for the profiler_config.json file modifications. After dynamic_profile is enabled for dynamic profiling, it records the modification time of the profiler_config.json file, whether the modifications take effect, and the end of the dynamic_profile process in real time. An example is shown below:

  1 2 3

  2024-08-21 15:51:46,392 [INFO] [2127856] _dynamic_profiler_monitor.py: Dynamic profiler process load json success 2024-08-21 15:51:58,406 [INFO] [2127856] _dynamic_profiler_monitor.py: Dynamic profiler process load json success 2024-08-21 15:58:16,795 [INFO] [2127856] _dynamic_profiler_monitor.py: Dynamic profiler process done

  File naming format: monitor_dp_{Operating system}_{monitor process ID}_{rank_id}.log.

• monitor_dp_ubuntu_xxxxxx.log.1: This is a log aging backup file. The storage limit for the monitor_dp_ubuntu_xxxxxx.log file is 200 KB. Once the limit is reached, the earliest log entries are moved to monitor_dp_ubuntu_xxxxxx.log.1. The storage limit for the monitor_dp_ubuntu_xxxxxx.log.1 file is also 200 KB, and once the limit is reached, the earliest log entries are deleted through aging.
• shm directory: To support Python 3.7, dynamic_profile will generate the shm directory in the environment. A binary file (DynamicProfileNpuShm+timestamp) is created in this directory to map shared memory. The file will be automatically cleaned up when the program ends normally. However, when the program is terminated using pkill, the program cannot release resources due to the abnormal termination, and you need to manually clean up this file. Otherwise, if dynamic_profile is started again within a short time (< 1h) using the same configuration path, it will cause dynamic_profile to fail. For Python 3.8 and above, the binary file (DynamicProfileNpuShm+timestamp) is stored in the /dev/shm directory. When the program is terminated using pkill, the file still needs to be manually cleaned up.