Profiling System Data of the AI processor

Function

msprof can profile system data of the AI processor, and automatically parse profile data and flush files to disks after profiling.

Command Format

msprof --output=<path> --sys-devices=<ID> --sys-period=<period> [options]

msprof [options] <app>

• For details about app, see app Parameters. For details about options, see Command-line Options.
• For Ascend EP, if you use the msprof CLI to collect profiling data for network-wide inference and the --llc-profiling, --sys-cpu-profiling, --sys-profiling, and --sys-pid-profiling options are enabled, only --sys-cpu-profiling generates TS CPU data, and no data will be generated for the remaining options. However, if no user program is passed in, all the preceding collection options generate data.
• For the models below, --sys-profiling, --sys-pid-profiling, and --sys-cpu-profiling options cannot be used to collect data of two devices that share the same OS. For example, if there are [0, 7] devices and they share OSs in groups of 0 and 1, 2 and 3, 4 and 5, and 6 and 7, respectively, --sys-devices cannot be set to 0 and 1, 2 and 3, 4 and 5, or 6 and 7 at the same time. It can be set to 0, 2, 4, and 6, or 1, 3, 5, and 7.
  • Atlas A3 training products/Atlas A3 inference products
  • Ascend 950PR/Ascend 950DT

Command-line Options

• --sys-period=<sys-period-value>: This parameter is mandatory if no user program is passed. It indicates the system sampling period (s). The value must be in the range (0, 30 × 24 × 3600]. This parameter will not be applied when a user program is passed.
• --sys-devices=<sys-devices-value>: This parameter is mandatory if no user program is passed. It indicates the device ID. The value can be all or multiple device IDs separated with commas (,). This parameter will not be applied when a user program is passed.
• --ai-core=<aicore-value>: (optional) controls AI Core data collection.
• --aic-mode=<aic-mode-value>: (optional) indicates the AI Core hardware collection type. The value can be task-based or sample-based. This option can only be used when --ai-core is set to on. In the task-based scenario, profile data is collected task by task. In the sample-based scenario, profile data is collected at a fixed interval.

  You are advised to use sample-based to collect AI processor system data. If this option is not set, sample-based is used by default.

• --aic-freq=<aic-freq-value>: (optional) indicates the sampling frequency in the sample-based scenario. The value ranges from 1 to 100, in Hz, and the default value is 100. This option can only be used when --ai-core is set to on.
• --aic-metrics=<aic-metrics-value>: (optional) indicates the AI Core metrics to profile. This option can only be used when --ai-core is set to on. Possible values are:
  • ArithmeticUtilization: percentage of the time consumed by computation instructions
  • PipeUtilization: total time consumed by computation and movement instructions and the percentage of the time consumed
  • Memory: memory read/write bandwidth rate
  • MemoryL0: L0 read/write bandwidth rate
  • MemoryUB: UB read/write bandwidth rate
  • ResourceConflictRatio: resource conflict ratio
  • L2Cache: L2 cache hit ratio

    Atlas inference products: not supported

  • PipelineExecuteUtilization: total time consumed by computation and movement instructions and the percentage of the time consumed

    Atlas inference products: not supported

    Atlas training products: not supported

    Atlas A2 training products/Atlas A2 inference products: not supported

    Atlas A3 training products/Atlas A3 inference products: not supported

    Ascend 950PR/Ascend 950DT: not supported

  • MemoryAccess:

    Atlas 200I/500 A2 inference products: not supported

    Atlas inference products: not supported

    Atlas training products: not supported

    Ascend 950PR/Ascend 950DT: not supported

  Default value:

  Atlas 200I/500 A2 inference products: PipelineExecuteUtilization

  Atlas inference products: PipeUtilization

  Atlas training products: PipeUtilization

  Atlas A2 training products/Atlas A2 inference products: PipeUtilization

  Atlas A3 training products/Atlas A3 inference products: PipeUtilization

  Ascend 950PR/Ascend 950DT: PipeUtilization

  You can specify the registers whose data is to be collected, for example, --aic-metrics=Custom:0x49,0x8,0x15,0x1b,0x64,0x10. The Custom field indicates the register type. It is set to specific register values in the range of [0x1, 0x7FFFFFFF]. Not all values have corresponding PMU registers. If the configured value does not have a corresponding PMU register, the profiling result may be 0. A maximum of eight registers can be configured. Separate them with commas (,). The register value can be in hexadecimal or decimal format.

• --sys-hardware-mem=<sys-hardware-mem-value>: (optional) controls the collection of the on-chip memory read/write rate, QoS transmission bandwidth, LLC L3 cache bandwidth, accelerator bandwidth, SoC transmission bandwidth, and component memory usage. The value can be on or off (default). The collected content varies slightly depending on the model.

  Component memory data can only be collected when AI job profile data collection is enabled (that is, when a user program is passed in).

  Collecting memory data in the environment where glibc (earlier than 2.34) is installed may trigger a known Bug 19329. You can solve this problem by upgrading the glibc version.

• --sys-hardware-mem-freq=<sys-hardware-mem-freq-value>: (optional) indicates the profiling frequency of --sys-hardware-mem. The value range is [1,100], in Hz, and the default value is 50.

  Ascend 950PR/Ascend 950DT: The maximum allowed profiling frequency for QoS and SoC data is 10,000 Hz. For other items, this limit remains 100 Hz, and any configuration higher than 100 Hz will automatically fall back to 100 Hz.

  This option can only be used when --sys-hardware-mem is set to on.

  For the following products, you are advised not to increase the profiling frequency after the profiling task is complete. Otherwise, SoC transmission bandwidth data may be lost.

  Atlas 200I/500 A2 inference products

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

• --llc-profiling=<llc-profiling-value>: (optional) indicates the LLC Profiling event. --sys-hardware-mem must be set to on. Possible values are:
  • read (default): read event, indicating the L3 cache read rate.
  • write: write event, indicating the L3 cache write rate.
• --sys-cpu-profiling=<sys-cpu-profiling-value>: (optional) profiles CPU (such as AI CPU and Ctrl CPU) data. The value can be on or off (default).
• --sys-cpu-freq=<sys-cpu-freq-value>: (optional) indicates the CPU profiling frequency. The value range is [1, 50], in Hz, and the default value is 50. This option can only be used when --sys-cpu-profiling is set to on.
• --sys-profiling=<sys-profiling-value>: (optional) profiles system CPU usage and system memory. The value can be on or off (default).

  After this option is used, the Profiling tool calls the perf tool on the device. The perf tool only collects profile data and cannot obtain other runtime information. The risk is low.

• --sys-sampling-freq=<sys-sampling-freq-value>: (optional) indicates the profiling frequency of system CPU usage and system memory. The value range is [1, 10], in Hz, and the default value is 10. This option can only be used when --sys-profiling is set to on.
• --sys-pid-profiling=<sys-pid-profiling-value>: (optional) profiles the CPU usage and memory of all processes. The value can be on or off (default).
• --sys-pid-sampling-freq=<sys-pid-sampling-freq-value>: (optional) indicates the profiling frequency of the CPU usage and memory of all processes. The value range is [1, 10], in Hz, and the default value is 10. This option can only be used when --sys-pid-profiling is set to on.
• --sys-io-profiling=<sys-io-profiling-value>: (optional) profiles NIC, RoCE, MAC, and UB bandwidth data. The value can be on or off (default).
  • Atlas 200I/500 A2 inference products: profiles NIC data only in RC scenarios. This option does not take effect in container scenarios.
  • Atlas inference products: This option is not supported.
  • Atlas training products: profiles NIC and RoCE data.
  • Atlas A2 training products/Atlas A2 inference products: profiles NIC, MAC, and RoCE data.
  • Atlas A3 training products/Atlas A3 inference products: profiles NIC, MAC, and RoCE data.
  • Ascend 950PR/Ascend 950DT: profiles UB bandwidth data.
• --sys-io-sampling-freq=<sys-io-sampling-freq-value>: (optional) indicates the profiling frequency of the NIC, RoCE, MAC, and UB bandwidth data. The value range is [1, 100], in Hz, and the default value is 100. This option can only be used when --sys-io-profiling is set to on.

  This option is not supported by the Atlas 200I/500 A2 inference products.

• --sys-interconnection-profiling=<sys-interconnection-profiling-value>: (optional) profiles HCCS, CCU bandwidth, PCIe data, inter-chip transmission bandwidth, SIO data, and UB bandwidth data. The collected content varies slightly depending on the model. The value can be on or off (default).
  • This option is not supported by the Atlas 200I/500 A2 inference products.
  • Atlas inference products: profiles PCIe data.
  • Atlas training products: profiles HCCS and PCIe data.
  • Atlas A2 training products/Atlas A2 inference products: profiles HCCS, PCIe data, and inter-chip transmission bandwidth.
  • Atlas A3 training products/Atlas A3 inference products: profiles HCCS, PCIe data, inter-chip transmission bandwidth, and SIO data.
  • Ascend 950PR/Ascend 950DT: profiles CCU bandwidth, PCIe data, inter-chip transmission bandwidth, UB bandwidth, and SIO data.
• --sys-interconnection-freq=<sys-interconnection-freq-value>: (optional) indicates the profiling frequency of the HCCS, CCU bandwidth, PCIe data, inter-chip transmission bandwidth, SIO data, and UB bandwidth data. The value range is [1, 50], in Hz, and the default value is 50.

  This option can only be used when --sys-interconnection-profiling is set to on.

  This option is not supported by the Atlas 200I/500 A2 inference products.

• --dvpp-profiling=<dvpp-profiling-value>: (optional) profiles the DVPP data. The value can be on or off (default).
• --dvpp-freq=<dvpp-freq-value>: (optional) indicates the DVPP profiling frequency. The value range is [1, 100], in Hz, and the default value is 50. This option can only be used when --dvpp-profiling is set to on
• --instr-profiling=<instr-profiling-value>: (optional) profiles the bandwidth and latency of AI Core (including AIC Core and AIV Core). The value can be on or off (default).

  This parameter is supported only by the following models:

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

  Ascend 950PR/Ascend 950DT

  Specific profile data can only be collected when AI task profile data collection is enabled (that is, when a user program is passed in) in single-operator API execution scenarios.

  For Atlas 350 Accelerator Card, the statistics may be inaccurate due to the long statistical period of the last instruction segment. You are advised to use msprof op for profiling.

  This option is mutually exclusive with --ascendcl, --model-execution, --runtime-api, --hccl, --task-time, --aicpu, --ai-core, --aic-mode, --aic-freq, --aic-metrics, and --l2 and cannot be executed at the same time.

• --instr-profiling-freq=<instr-profiling-freq-value>: (optional) indicates the profiling interval for the bandwidth and latency of the AI Core (including AIC Core and AIV Core). The value range is [300, 30000], in cycles, and the default value is 1000. Actual AI Core bandwidth and latency profiling frequency = Processor operating frequency/Value of this option. Suppose that the AI Core operating frequency is 5000 Hz and the value of this option is 1000. The profiling frequency is 5 Hz, which is 5 times per second.

  This option can only be used when --instr-profiling is set to on.

  This parameter is supported only by the following models:

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

Example

msprof --output=/home/projects/output --sys-devices=<ID> --sys-period=<period> --ai-core=on --sys-hardware-mem=on --sys-cpu-profiling=on --sys-profiling=on --sys-pid-profiling=on --dvpp-profiling=on

In the Ascend EP scenario, the PROF_XXX directory is generated in the directory specified by --output to store the automatically parsed profile data. For details about the result files, see Profile Data File References.

In the Ascend RC scenario, the PROF_XXX directory is generated in the directory specified by --output. Since the files in this directory cannot be viewed without being parsed, you need to upload the PROF_XXX directory to the development environment for data parsing. For details, see Using the msprof Command to Parse, Query, and Export Profile Data.