Profiling Data Collection

Collective communication is a global collaborative behavior in a communicator, and using profiling data of only one rank is difficult to analyze the performance problems of collective communication. To pinpoint performance bottlenecks accurately, you need to collect profiling data from all ranks. Currently, profiling data can be collected in either of the following ways:

• Method 1: Collect profiling data by referring to Performance Tuning Tool.
• Method 2: Use HCCL Performance Tester to collect profiling data and test the performance by referring to HCCL Performance Tester.
  To collect profiling data using HCCL Performance Tester, perform the following steps:

  # 1 indicates that profiling is enabled, and 0 indicates that profiling is disabled. The default value is 0. If profiling is enabled, profiling data is collected when HCCL Performance Tester is executed.
  export HCCL_TEST_PROFILING=1
  # Specify the path for storing profiling data. The default path is /var/log/npu/profiling.
  export HCCL_TEST_PROFILING_PATH=/home/profiling

  If HCCL_TEST_PROFILING is enabled, profiling data is generated in the directory specified by HCCL_TEST_PROFILING_PATH after HCCL Performance Tester is executed. For details about how to parse the profiling data, see Using the msprof Command to Parse, Query, and Export the Profile Data in Performance Tuning Tool.