Profiling

The profiler supports dynamic start and stop at runtime. The procedure is as follows:

Configure the environment variable to specify the profiling configuration file ms_service_profiler_config.json.
```
export SERVICE_PROF_CONFIG_PATH="./ms_service_profiler_config.json"
```
- If the .json configuration file does not exist in the path configured for the environment variable, a .json file with default configurations is automatically created in the path and enable is set to 0. You need to perform 3 after running the MindIE Motor service and set enable to 1 to enable the profiling task.
- If a .json file with the same name already exists in the path configured for the environment variable, no .json file will be created.
Run the MindIE Motor service.
Enable the profiling task.
Open another CLI window, and modify the enable field in the ms_service_profiler_config.json configuration file to enable or disable profiling in real time. Logs are generated when the profiling is enabled or disabled. For details, see Dynamic Start and Stop.

After the profiling is complete, the profile data is flushed to the path specified by the prof_dir parameter in the ms_service_profiler_config.json file.

You can use the Samba tool to share the configuration file for profiling across multiple nodes and ranks. In multi-node and multi-rank setups, follow the same profiling steps mentioned earlier but launch MindIE Motor on every node. Samba is a third-party tool. Search for its usage guide online or try alternative tools for configuring shared directories.
The acl_task_time switch of the profiler conflicts with the dynamic profiling function of the msprof tool. You are advised not to use them at the same time. For details about the dynamic profiling function of the msprof tool, see Profiling Instructions.

Dynamic Start and Stop

The dynamic start and stop function allows you to start or stop profiling whenever needed.

There are three main scenarios as follows:

Enable the function. The enable field is set to 0 in the .json configuration file before starting MindIE Motor. Change enable to 1 during service running, and the log prints a message indicating that the profiling function is enabled.
Disable the function. The enable field is set to 1 in the .json configuration file before starting MindIE Motor. Change enable to 0 during service running, and the log prints a message indicating that the profiling function is disabled.
The .json configuration file is updated but the enable field remains unchanged. In this case, the profiling function status remains unchanged and the log prints the following message:

Log Description

Logs are printed during profiling to indicate the status of the profiling process. You can configure the PROF_LOG_LEVEL environment variable to control the log printing as follows:

The PROF_LOG_LEVEL environment variable is used to configure the profiling log level. The following is an example:

export PROF_LOG_LEVEL=INFO

The log level can be set as follows (the default level is INFO):

INFO: information such as whether to enable profiling and the data flush path.

Example:

[msservice_profiler] [PID:52856] [INFO] [ReadEnable:306] profile enable_: true
[msservice_profiler] [PID:52856] [INFO] [ReadAclTaskTime:335] profile enableAclTaskTime_: false
[msservice_profiler] [PID:52856] [INFO] [StartProfiler:661] prof path: ./log/0423-0852/

DEBUG: detailed log information, including the configuration file path, and whether to enable NPU and CPU profiling.

Example:

[msservice_profiler] [PID:82231] [DEBUG] [ReadConfig:275] SERVICE_PROF_CONFIG_PATH : prof.json
[msservice_profiler] [PID:82231] [DEBUG] [ReadLevel:386] profiler_level: 20
[msservice_profiler] [PID:82231] [DEBUG] [ReadHostConfig:510] host_system_usage_freq Disabled
[msservice_profiler] [PID:82231] [DEBUG] [ReadNpuConfig:541] npu_memory_usage_freq Disabled

WARNING: warnings about incorrect parameter settings and dynamic library loading failures.

[msservice_profiler] [PID:43982] [WARNING] [ReadEnable:323] enable value is not an integer, will set false.
[msservice_profiler] [PID:43984] [WARNING] [ReadEnable:323] enable value is not an integer, will set false.
[msservice_profiler] [PID:43993] [WARNING] [ReadEnable:323] enable value is not an integer, will set false.
[msservice_profiler] [PID:44002] [WARNING] [ReadEnable:323] enable value is not an integer, will set false.

ERROR: errors.

Example:

[msservice_profiler] [PID:87888] [ERROR] [StartProfiler:677] create path(./log/0423-1007/) failed

Parent topic: Data Profiling