文档
下载

PROFILING_OPTIONS

Description

Sets profiling configuration options in training or online inference scenarios.

  • output: path for storing profiling result files. Both absolute path and relative path (relative to the path where the command is run) are supported. The path cannot contain the following special characters: "\n", "\f", "\r", "\b", "\t", "\v", and "\u007F".
    • An absolute path starts with a slash (/), for example, /home/output.
    • A relative path starts with a directory name, for example, output.
    • It takes precedence over ASCEND_WORK_PATH.
    • This path does not need to be created in advance because it is automatically created during collection.
  • storage_limit: maximum size of files that can be stored in a specified disk directory. If the size of profile data files in the disk is about to use up the maximum storage space specified by this option or the total remaining disk space is about to be used up (remaining space ≤ 20 MB), the earliest files in the disk are aged and deleted.

    The value range is [200, 4294967295], and the unit is MB. The unit must be included when you set this parameter, for example, 200 MB.

    If this option is not set and the available disk space is less than 20 MB before data collection, data is not flushed to the disk.

  • training_trace: iteration tracing switch. Collects software profile data of a training job and the AI Software Stack to profile the training job. Focuses on forward and backward propagation, and gradient aggregation and update. This option must be set to on when the forward and backward propagation operator data is collected.
  • task_trace and task_time: switches that control collection of the operator dispatch and execution durations. Related duration data must be output to the task_time, op_summary, and op_statistic files. Possible configuration values are as follows:
    • on: switch on. This is the default value, delivering the same effect as l1.
    • off: switch off.
    • l0: collects operator delivery and execution duration data. Compared with l1, l0 does not collect basic operator information, so the performance overhead during collection is smaller, and this enables more accurate collection of statistics on time duration data.
    • l1: collects operator delivery and execution duration data, as well as basic operator information, to provide more comprehensive performance analysis data.

    When Profiling is enabled to collect training data, task_trace and training_trace must be set to on.

  • ge_api: switch that controls collection of the time consumption data of dynamic-shape operators in the host scheduling phase. Possible values are as follows:
    • off: switch off. The default value is off.
    • l0: collects the time consumption data of dynamic-shape operators in the main host scheduling phase to facilitate accurate statistics.
    • l1: collects finer-grained time consumption data of dynamic-shape operators in the host scheduling phase to provide more comprehensive performance analysis data.
  • hccl: communication data collection switch, either on or off (default).

    This switch will be deprecated in later versions. To control data collection, use task_time.

  • aicpu: whether to collect details about the AI CPU operator, such as the operator execution time and data copy time. The value can be on or off (default).
  • fp_point: start point of the forward propagated operator in iteration traces, to record the start timestamp of forward propagation. Set the value to the name of the top operator in forward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain the name. Alternatively, you can leave this option empty (for example, "fp_point":""), and the system will automatically identify the start point of the forward propagated operator.
  • bp_point: end point of the backward propagated operator in iteration traces, to record the end timestamp of backward propagation. bp_point and fp_point are used to compute the time used by forward and backward propagation. Set the value to the name of the bottom operator in backward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain the name. Alternatively, you can leave this option empty (for example, "bp_point":""), and the system will automatically identify the end point of the backward propagated operator.
  • aic_metrics: AI Core metric to profile. The options are as follows:
    • ArithmeticUtilization: arithmetic utilization ratio.
    • PipeUtilization (default): ratio of time taken by the compute units to that of MTEs.
    • Memory: ratio of external memory read/write instructions.
    • MemoryL0: ratio of internal memory L0 read/write instructions.
    • MemoryUB: ratio of internal memory UB read/write instructions.
    • ResourceConflictRatio: ratio of pipeline queue instructions.
    • L2Cache: read/write L2 cache hits and re-allocations after cache misses

      Atlas inference product: This parameter is not supported.

      Atlas training product: This parameter is not supported.

    • MemoryAccess: bandwidth of the operator's memory access on cores.

      Atlas inference product: This parameter is not supported.

      Atlas training product: This parameter is not supported.

      Atlas 350 Accelerator Card: This parameter is not supported.

    The registers whose data is to be collected can be customized, for example, "aic_metrics":"Custom:0x49,0x8,0x15,0x1b,0x64,0x10".
    • The Custom field indicates the customization type. It is set to specific register values in the range of [0x1, 0x6E].
    • A maximum of eight registers can be configured, which are separated with commas (,).
    • The register value can be in hexadecimal or decimal format.
  • l2: switch that controls L2 cache and TLB page table cache hit ratio, either on or off (default).
    • Atlas inference product: profiles the L2 cache hit ratio.
    • Atlas training product: profiles the L2 cache hit ratio.
    • Atlas A2 training product/Atlas A2 inference product: profiles the L2 cache hit ratio and TLB page table cache hit ratio. aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 cache from the AI Core.
    • Atlas A3 training product/Atlas A3 inference product: profiles the L2 cache hit ratio and TLB page table cache hit ratio. aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 cache from the AI Core.
    • Atlas 350 Accelerator Card: profiles the L2 cache hit ratio and TLB page table cache hit ratio. aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 cache from the AI Core.
  • msproftx: switch that controls the msproftx user and upper-layer framework program to output profile data, either on or off (default).

    Add the following mstx API or msproftx API to the application script. The mstx API is recommended.

  • runtime_api: runtime API data profiling switch, either on or off (default). You can profile runtime API data, including the synchronous/asynchronous memory replication latencies between the host and device and between devices.
  • sys_hardware_mem_freq: switch that controls the collection of the on-chip memory, QoS transmission bandwidth, LLC L3 cache bandwidth, accelerator bandwidth, SoC transmission bandwidth, and component memory usage. The collected items vary slightly depending on the product. The value range is [1, 100], in Hz.

    Atlas 350 Accelerator Card: The maximum allowed collection frequency for QoS and SoC data is 10,000 Hz. For other collection items, this limit remains 100 Hz, and any configuration higher than 100 Hz will automatically fall back to 100 Hz.

    Sampling memory data in the environment where glibc (2.34 or earlier) is installed may trigger a known Bug 19329. This problem can be solved by upgrading the glibc version.

    For the following products, you are advised not to increase the profiling frequency after the profiling task is complete. Otherwise, SoC transmission bandwidth data may be lost.

    Atlas 200I/500 A2 inference product

    Atlas A2 training product/Atlas A2 inference product

    Atlas A3 training product/Atlas A3 inference product

  • llc_profiling: LLC events to profile. Possible values are as follows:
    • read (default): read events, that is, the L3 cache read rate.
    • write: write event, indicating the L3 cache write rate
  • sys_io_sampling_freq: NIC, UB bandwidth, and RoCE profiling frequency. The value range is [1, 100], in Hz.

    Atlas inference product: This parameter is not supported.

    Atlas A2 training product/Atlas A2 inference product: Allowed for NIC and RoCE collection

    Atlas A3 training product/Atlas A3 inference product: Allowed for NIC and RoCE collection

    Atlas 350 Accelerator Card: Allowed for UB bandwidth data profiling

  • sys_interconnection_freq: frequency of collecting collective communication bandwidth data (HCCS), collective communication hardware accelerator unit (CCU) bandwidth data, SIO data, PCIe data, UB bandwidth data, and inter-chip transmission bandwidth information. The value range is [1, 50], in Hz.
    • Atlas training product: Allowed for HCCS and PCIe data collection
    • Atlas A2 training product/Atlas A2 inference product: Allowed for HCCS, PCIe data, and inter-chip transmission bandwidth information collection
    • Atlas A3 training product/Atlas A3 inference product: Allowed for HCCS, PCIe data, inter-chip transmission bandwidth information, and SIO data collection
    • Atlas 350 Accelerator Card: Allowed for PCIe data, inter-chip transmission bandwidth information, CCU bandwidth data, SIO data, and UB bandwidth data collection
  • dvpp_freq: DVPP collection frequency. The value range is [1, 100], in Hz.
  • instr_profiling: AI Core and AI Vector bandwidth and latency collection switch. The value can be on or off (default).
    • Atlas training product: This function is not supported.
    • Atlas A2 training product/Atlas A2 inference product: This switch is not supported. This function is controlled through instr_profiling_freq.
    • Atlas A3 training product/Atlas A3 inference product: This switch is not supported. This function is controlled through instr_profiling_freq.
    • Atlas 350 Accelerator Card: This function is supported, but the statistics may be inaccurate due to the long statistical period of the last instruction segment. You are advised to use msprof op for profiling.
  • instr_profiling_freq: AI Core and AI Vector bandwidth and latency collection switch. If the collection frequency is configured, the related collection capability is enabled. The value range is [300, 30000]. The unit is Hz.
    • Atlas training product: This function is not supported.
    • Atlas A2 training product/Atlas A2 inference product: This function is supported. However, instr_profiling_freq is mutually exclusive with training_trace, task_trace, hccl, aicpu, fp_point, bp_point, aic_metrics, l2, task_time, and runtime_api, so they cannot be executed at the same time.
    • Atlas A3 training product/Atlas A3 inference product: This function is supported. However, instr_profiling_freq is mutually exclusive with training_trace, task_trace, hccl, aicpu, fp_point, bp_point, aic_metrics, l2, task_time, and runtime_api, so they cannot be executed at the same time.
    • Atlas 350 Accelerator Card: This switch is not supported. This function is controlled through instr_profiling.
  • host_sys: switch for collecting host profile data. You can select one or more options and separate them with commas (,), for example, "host_sys": "cpu,mem".
    • cpu: process CPU utilization
    • mem: process memory utilization
    • disk: process disk I/O utilization To collect host-side disk profile data, install the third-party open-source tool iotop. To collect osrt profile data, install the third-party open-source tools perf and ltrace.
    • network: network I/O utilization
    • osrt: process syscall and pthreadcall
  • host_sys_usage: Host-side system and process CPU and memory data collection option, selected from cpu and mem. You can select one or more options and separate them with commas (,).
  • host_sys_usage_freq: Host-side system and process CPU and memory data collection frequency. The value range is [1, 50] and the default value is 50. The unit is Hz.
  • fp_point and bp_point require manual configuration only in the dynamic shape scenario. In the dynamic shape scenario, fp_point and bp_point must be configured manually.
  • Online inference supports task_trace and aicpu but does not support training_trace.

Example

export PROFILING_OPTIONS='{"output":"/tmp/profiling","training_trace":"on","task_trace":"on","fp_point":"","bp_point":"","aic_metrics":"PipeUtilization"}'

Restrictions

  • This environment variable applies only to TensorFlow training or online inference cases.
  • Before using this environment variable, use PROFILING_MODE to enable the profiling function.

Applicability

Atlas training product

Atlas inference product

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card

Parent topic: Profiling
我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情