Profiling AI Tasks When They Are Running

Function

msprof supports profiling of AI tasks when they are running. After the profiling, it automatically parses the profile data and flush corresponding files to disks.

Command Format

msprof [options] <app>

app is mandatory. For details about the parameters, see app Parameters. For details about options, see Command-line Options.

Command-line Options

• --ascendcl=<ascendcl-value>: (optional) controls the profiling of acl APIs. The value can be on (default) or off. You can profile acl API data, including the synchronous/asynchronous memory copy latencies between the host and devices and between devices.
• --model-execution=<model-execution-value>: (optional) controls the profiling of GE model execution. The value can be on or off (default). This option will be deprecated in later versions. Use --task-time instead.
• --runtime-api=<runtime-api-value>: (optional) controls the profiling of Runtime APIs. The value can be on or off (default). You can profile Runtime API data, including the synchronous/asynchronous memory copy latencies between the host and device and between devices.
• --hccl=<hccl-value>: (optional) controls the communication data collection. The value can be on or off (default). The data is generated only in multi-card, multi-node, or cluster scenarios. This option will be deprecated in later versions. Use --task-time instead.
• --task-time=<task-time-value>: (optional) controls the profiling of the operator delivery and execution durations. Related duration data must be output to files such as task_time, op_summary, and op_statistic. Possible configuration values are as follows:
  • l0: collects operator delivery and execution duration data. Compared with l1, l0 does not collect basic operator information, so the performance overhead during profiling is smaller, and this enables more accurate profiling on time consumption data.
  • l1: collects operator delivery and execution duration data, as well as basic operator information, to provide more comprehensive performance analysis data. This option supports collecting the collective communication operator data.
  • l2: collects data on operator delivery duration, operator execution duration, and basic operator information (including attr information) to provide more comprehensive performance analysis data. This option supports collecting the collective communication operator data.
  • l3: collects the profile data of the PyPTO operator. This feature is for trial use and may be changed in later versions. It is not available in commercial products.
  • on (default): This option is enabled. This value has the same effect as l1.
  • off: This option is disabled.
• --aicpu=<aicpu-value>: (optional) collects detailed information about AI CPU operators, such as the computation time and data copy time. The value can be on or off (default).
• --ai-core=<aicore-value>: (optional) controls AI Core data collection. The value can be on or off. When --task-time is set to on or l1, the default value is on. When --task-time is set to off or l0, the default value is off.
• --aic-mode=<aic-mode-value>: (optional) indicates the AI Core hardware collection type. The value can be task-based or sample-based. This option can only be used when --ai-core is set to on. In the task-based scenario, profiling is performed by task. In the sample-based scenario, profiling is performed at a fixed interval.

  You are advised to use task-based to collect AI task profile data. If this option is not set, task-based is used by default.

• --aic-freq=<aic-freq-value>: (optional) indicates the sampling frequency in the sample-based scenario. The value ranges from 1 to 100, in Hz, and the default value is 100. This option can only be used when --ai-core is set to on.
• --aic-metrics=<aic-metrics-value>: (optional) indicates the AI Core metrics to profile. This option can only be used when --ai-core is set to on. Possible values are:
  • ArithmeticUtilization: percentage of the time consumed by computation instructions
  • PipeUtilization: total time consumed by computation and movement instructions and the percentage of the time consumed
  • Memory: memory read/write bandwidth rate
  • MemoryL0: L0 read/write bandwidth rate
  • MemoryUB: UB read/write bandwidth rate
  • ResourceConflictRatio: resource conflict ratio
  • L2Cache: L2 cache hit ratio

    Atlas inference products: not supported

  • PipelineExecuteUtilization: total time consumed by computation and movement instructions and the percentage of the time consumed

    Atlas inference products: not supported

    Atlas training products: not supported

    Atlas A2 training products/Atlas A2 inference products: not supported

    Atlas A3 training products/Atlas A3 inference products: not supported

    Ascend 950PR/Ascend 950DT: not supported

  • MemoryAccess:

    Atlas 200I/500 A2 inference products: not supported

    Atlas inference products: not supported

    Atlas training products: not supported

    Ascend 950PR/Ascend 950DT: not supported

  Default value:

  Atlas 200I/500 A2 inference products: PipelineExecuteUtilization

  Atlas inference products: PipeUtilization

  Atlas training products: PipeUtilization

  Atlas A2 training products/Atlas A2 inference products: PipeUtilization

  Atlas A3 training products/Atlas A3 inference products: PipeUtilization

  Ascend 950PR/Ascend 950DT: PipeUtilization

  You can specify the registers whose data is to be collected, for example, --aic-metrics=Custom:0x49,0x8,0x15,0x1b,0x64,0x10. The Custom field indicates the register type. It is set to specific register values in the range of [0x1, 0x7FFFFFFF]. Not all values have corresponding PMU registers. If the configured value does not have a corresponding PMU register, the profiling result may be 0. A maximum of eight registers can be configured. Separate them with commas (,). The register value can be in hexadecimal or decimal format.

• --sys-hardware-mem=<sys-hardware-mem-value>: (optional) controls the collection of the on-chip memory read/write rate, QoS transmission bandwidth, LLC L3 cache bandwidth, accelerator bandwidth, SoC transmission bandwidth, and component memory usage. The value can be on or off (default). The collected content varies slightly depending on the model.

  Collecting memory data in the environment where glibc (earlier than 2.34) is installed may trigger a known Bug 19329. You can solve this problem by upgrading the glibc version.

• --sys-hardware-mem-freq=<sys-hardware-mem-freq-value>: (optional) indicates the profiling frequency of --sys-hardware-mem. The value range is [1,100], in Hz, and the default value is 50.

  Ascend 950PR/Ascend 950DT: The maximum allowed profiling frequency for QoS and SoC data is 10,000 Hz. For other items, this limit remains 100 Hz, and any configuration higher than 100 Hz will automatically fall back to 100 Hz.

  This option can only be used when --sys-hardware-mem is set to on.

  For the following products, you are advised not to increase the profiling frequency after the profiling task is complete. Otherwise, SoC transmission bandwidth data may be lost.

  Atlas 200I/500 A2 inference products

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

• --l2=<l2-value>: (optional) collects the hit rates of the L2 cache and TLB page table. The value can be on or off (default).
  If profiling is enabled during model execution in the aclgraph scenario, this profiling item does not take effect.

  • Atlas A2 training product/Atlas A2 inference product: --aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 from the AI Core.
  • Atlas A3 training product/Atlas A3 inference product: --aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 from the AI Core.
• --ge-api=<ge-api-value>: (optional) collects the time consumption data of dynamic shape operators in the host scheduling phase. Related data is generated in the msprof_*.json and api_statistic_*.csv files. Possible values are as follows:
  • off: (default): This option is disabled.
  • l0: collects the time consumption data of dynamic-shape operators in the main host scheduling phase to facilitate accurate statistics.
  • l1: profiles finer-grained time consumption data of dynamic-shape operators in the host scheduling phase to provide more comprehensive profile data.
• --task-memory=<task-memory-value>: (optional) collects CANN operator memory usage for optimization. The options are as follows:
  • on: This option is enabled.
  • off (default): This option is disabled.

  In the graph-mode single-operator scenario, the operator memory size and lifecycle information are collected based on the GE component and operator dimensions (the GE component memory is not collected in the single-operator API execution scenario). In the static-graph and static-subgraph scenarios, the operator memory size and lifecycle information is collected based on the operator dimension.

• --task-block=<task-block-value>: (optional) indicates the profiling at the block level.

  This option is supported only by the following models:

  Ascend 950PR/Ascend 950DT: The value can be on or off (default).

Example

msprof --output=/home/projects/output --ascendcl=on --runtime-api=on --task-time=on --aicpu=on --ai-core=on /home/projects/MyApp/out/main

In the Ascend EP scenario, the PROF_XXX directory is generated in the directory specified by --output to store the automatically parsed profile data. For details about the result files, see Profile Data File References.

In the Ascend RC scenario, the PROF_XXX directory is generated in the directory specified by --output. Since the files in this directory cannot be viewed without being parsed, you need to upload the PROF_XXX directory to the development environment for data parsing. For details, see Using the msprof Command to Parse, Query, and Export Profile Data.