Before You Start

Environment Preparation

1. Install CANN packages based on the actual scenario and configure CANN environment variables. For details, see CANN Software Installation.

   In the Ascend EP scenario, the msprof tool is saved in ${INSTALL_DIR}/tools/profiler/bin. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.

   In the Ascend RC scenario, the msprof tool is saved in /var.

2. Set public environment variables.

   After the CANN software is installed, log in to the environment as the CANN operating user and run the source ${INSTALL_DIR}/set_env.sh command to set environment variables. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.

3. Set Python environment variables.
   If there are multiple Python 3 series versions, specify your Python installation path as needed. The following takes Python 3.7.5 as an example.

   export PATH=/usr/local/python3.7.5/bin:$PATH
   # Set the library path for Python 3.7.5.
   export LD_LIBRARY_PATH=/usr/local/python3.7.5/lib:$LD_LIBRARY_PATH

The preceding environment variables take effect only in the current window. You can write the preceding commands to the ~/.bashrc file to make them take effect permanently. The operations are as follows:

1. Run the vi ~/.bashrc command in any directory as the installation user and append the preceding commands to the file.
2. Run the :wq! command to save the file and exit.
3. Run the source ~/.bashrc command to make the environment variables take effect.

Restrictions

Before using the tool, understand the following restrictions:

• Permission Restrictions
  • Adhere to the principle of least privilege. Specifically, eliminate world-writable permissions such as 666 and 777.
  • Before using the profiler, ensure that the execution user's umask is 0027 or more restrictive. Otherwise, the generated profile data directories and files will have excessive permissions.
  • To ensure security and adhere to the principle of least privilege, you are advised to run this tool as a standard user rather than a high-privilege user (such as root).
  • The tool is a development and debugging tool and should not be used in the production environment.
  • This tool depends on CANN and the corresponding driver software package. Before using this tool, install CANN and the required driver software package, and run the source command to execute the set_env.sh environment variable file of CANN. To ensure security, do not modify the environment variables in set_env.sh after running the source command.
  • Ensure that the profile data is stored in the current user directory that does not contain soft links. Otherwise, security problems may occur.
• Execution Restrictions
  • Simultaneously initiating multiple profiling tasks on the same device is not supported. Additionally, two or more profilers cannot be started at the same time.
  • Avoid using the profiling function together with the dump function. The dump operation could affect the system performance. If the profiling and dump functions are both enabled, the profile data metrics will be inaccurate. Therefore, you should disable data dump before starting data profiling.
• Data Flush Restrictions
  • The recommended duration for data profiling is within 5 minutes. Reserve memory and disk space that is at least 20 times the size of the raw profile data. The size of raw data refers to the total amount of data saved in the data directory after profiling.
  • For data profiling and flushing in a single task when all profiling items are enabled, ensure that the disk read/write speed meets the requirements.
    • If your inference task runs on a single device, ensure that the disk read/write speed is 50 MB/s or higher.
    • If your training job runs on a single device, ensure that the disk read/write speed is 60 MB/s or higher.
    • If your job runs on multiple devices, the disk read/write speed must be greater than or equal to that of a single device multiplied by the number of devices.
  • Profile data cannot be flushed to a disk if the disk is full during profiling. Therefore, it is necessary to leave enough space on the disk. You can configure --storage-limit to prevent the disk space from being used up by the profile data.
  • During profile data parsing, if the configured disk or user directory space is full, the parsing fails or the file cannot be flushed to the disk. In this case, you need to clear the disk or user directory space.
• Compatibility and Scenario Restrictions
  • Python 3.7.5 or later is required.
  • The application project development must comply with Application Development (C&C++). To obtain complete profile data, call aclInit() for initialization and call aclFinalize() for deinitialization.
    

    If the application has called aclInit() but not aclFinalize(), the profiling process will be terminated abnormally, resulting in incomplete profile data. The data profiled within the last second may be lost due to delayed flushing, but the lost data is no more than 2 MB, and this does not affect the analysis of the flushed profile data.

  • If you use the msprof CLI for data profiling of an application project developed with pyACL APIs, the Python script cannot access files in relative paths. Otherwise, errors will be reported during profiling.
  • Ascend virtual instance: Profiling SoC system data (excluding the NPU memory usage) from containers or VMs is not allowed.