Collecting AI Core Error Information

Information Types

Note: When collecting logs, you need to collect logs generated around the time when AI Core errors occur. If irrelevant information or other error information in the old logs is collected, the msaicerr tool may report errors when analyzing the AI Core errors.

Collecting AI Core errors involves obtaining logs and setting environment variables. This section uses the default log paths to describe how to obtain various logs and takes examples to describe how to set environment variables.

• For details about log levels, log paths, and log files, see Log Reference.
• For details about the environment variables and their restrictions, see Environment Variables.

Determining Whether Service Re-run Is Required Before Collecting Information

Note: If the time of the AI Core error in the log is long after the occurrence time of the error, the old log may have been overwritten or deleted. In this case, you need to re-run the service to collect the fault information again.

Description: If you are sensitive to performance but have sufficient disk resources, and want to automatically generate an exception dump file to analyze AI Core errors without stopping services, you can call the aclInit API to enable the dump configuration of the exception operator (that is, configure the dump_scene parameter in the JSON file). In this case, the performance of most networks deteriorates by less than 1%. However, after the configuration is enabled, dump files generated for multiple AI Core errors consume disk space. You need to clear historical dump data. For details about the API, see aclInit.

Collection Methods

When locating AI Core errors, you need to collect fault information in advance. The following two methods are available.

Manual Collection

• When service re-run is not required
  1. Create an empty directory (for example, aic_err_info) for saving all the collected AI Core error information.

     mkdir ${HOME}/aic_err_info

  2. Collect application logs, trace logs, exception dump files, and abnormal operator compilation information, and move them to the ${HOME}/aic_err_info directory for unified management.

     Obtain the information from the directories specified by the following environment variables:

     • Obtain application logs from the path specified by the ASCEND_PROCESS_LOG_PATH environment variable. For details about the ASCEND_PROCESS_LOG_PATH environment variable, see Logs > ASCEND_PROCESS_LOG_PATH.
     • Obtain application logs, trace logs, and exception dump files from the path specified by the ASCEND_WORK_PATH environment variable. For details about the ASCEND_WORK_PATH environment variable, see Installation > ASCEND_WORK_PATH.
     • If the two environment variables exist at the same time, obtain application logs from the path specified by the ASCEND_PROCESS_LOG_PATH environment variable, and obtain trace logs and exception dump files from the path specified by the ASCEND_WORK_PATH environment variable.

     If the corresponding environment variable is not configured in the current environment, you can obtain the information from the corresponding default storage directories.

     • The default directory for storing application logs is ${HOME}/ascend/log on the host server.
     • The default directory for storing trace logs is $HOME/ascend/atrace/ on the host server.
     • The default directory for storing exception dump files and abnormal operator compilation information is the extra-info/data-dump/{device_id} directory of the current path where the inference application or training script is executed. This directory contains the input and output data files exception_info.{stream_id}.{task_id}.{timestamp} related to AI Core errors and the abnormal operator compilation information (.o and .json files).
     

     • If no abnormal operator compilation information is obtained based on the preceding environment variables or default directory, obtain the information by referring to Manually Collecting Operator Compilation Information (.o and .json Files).
     • The trace logs on the host server cannot be automatically cleared and may occupy a large amount of disk space. If the memory space is limited, you can run the following command to copy the trace logs of a specified process in the $HOME/ascend/atrace/ directory as required: cp -rf $HOME/ascend/atrace/trace_{PID}* aic_err_info/.
     • The dump files collected here cannot be viewed in a text tool. To view the dump file content, convert the dump files to NumPy files and then view the NumPy files through Python. For details about the conversion procedure, see Viewing Dump Data Files in Accuracy Analyzer. The following is an example of the conversion result:

       # Example of the dump file before conversion
       aclnnMatmul_4886226_L0.MatMulCommon.3975510.1717053072064889
       # Examples of the npy files after conversion
       aclnnMatmul_4886226_L0.MatMulCommon.3975510.1717053072064889.input.0.npy
       aclnnMatmul_4886226_L0.MatMulCommon.3975510.1717053072064889.input.1.npy
       aclnnMatmul_4886226_L0.MatMulCommon.3975510.1717053072064889.output.0.npy

  3. Collect device-side system logs and other maintenance and test information, including slogs, syslogs, and black box logs.
     Run the msnpureport tool as the root user on the host, export the logs to the host, and move the logs to the aic_err_info directory.

     # Run the msnpureport command in a directory (for example, ${HOME}/ascend/report) on which you possess both the read and write permissions on the host.
     msnpureport -f
     
     # Move log files to the aic_err_info directory.
     mv ${HOME}/ascend/report aic_err_info/ 

  4. Move the information in the ${HOME}/aic_err_info directory to another directory. The directory name can contain a timestamp so that you can manage the collected fault information by time.

     mv ${HOME}/aic_err_info  ${HOME}/aic_err_info_timestamp

• When service re-run is required
  1. Create an empty directory (for example, aic_err_info) for saving all the collected AI Core error information.

     mkdir ${HOME}/aic_err_info

  2. Set environment variables.
     The following is an example of setting the environment variables:

     # By default, the plog and atrace directories are generated in the directory specified by ASCEND_WORK_PATH to store application logs and trace logs, respectively.
     export ASCEND_WORK_PATH=${HOME}/aic_err_info
     
     # Specify the directory for storing application logs. The directory specified by the ASCEND_PROCESS_LOG_PATH environment variable has a higher priority for storing application logs.
     export ASCEND_PROCESS_LOG_PATH=${HOME}/aic_err_info/plog
     
     # By default, the extra-info directory is generated in the directory specified by NPU_COLLECT_PATH. The data-dump subdirectory stores the exception dump files and abnormal operator compilation information.
     export NPU_COLLECT_PATH=${HOME}/aic_err_info/

     For details about the environment variables and their restrictions, see the following sections in Environment Variables.

     • Installation > ASCEND_WORK_PATH
     • Logs > ASCEND_PROCESS_LOG_PATH
     • Fault Information Collection > NPU_COLLECT_PATH
  3. Re-run the service, that is, run the service program of the user again. Obtain the application logs, trace logs, exception dump files, and abnormal operator compilation information from the path specified by the environment variables.

     Check whether the operator compilation information (.o and .json files) exists in the path specified by the NPU_COLLECT_PATH environment variable. If not, search for the information in the default CANN installation path /usr/local/Ascend/cann and then in the default directory ${HOME}/atc_data. For details, see Manually Collecting Operator Compilation Information (.o and .json Files).

  4. Collect device-side system logs and other maintenance and test information. The collection method is the same as that in the scenario where service re-run is not required. For details, see 3.
  5. Move the information in the ${HOME}/aic_err_info directory to another directory. The directory name can contain a timestamp so that you can manage the information by time.

     mv ${HOME}/aic_err_info  ${HOME}/aic_err_info_timestamp

Automatic Collection

For details about the restrictions on using the asys tool, see Functions and Restrictions of the asys Tool. Before using the asys tool, install and configure it. For details, see the prerequisites in Preparing the asys Tool Environment.

• When service re-run is not required, run the asys collect command to collect fault information.

  asys collect --output=path

  output indicates the directory where the collected information is stored. For details about the parameters and restrictions, see Fault Information Collection.

• When service re-run is required, run the asys launch command to concurrently re-run the service and collect fault information.

  asys launch --task="sh ../app_run.sh" --output=path

  task indicates the task to be resumed, and output indicates the directory for storing collected information. For details about the parameters and restrictions, see Service Re-run And Fault Information Collection.

  Note: In an offline inference scenario, to rebuild a model (for example, using the ATC tool to convert the model), run the asys launch command to re-run the model building task, and then run the asys launch command to re-run the inference service based on the recompiled model. Additionally, place the maintenance and test information collected during model building and inference in the same directory, for example, $HOME/asys_output.

After information is collected using the tool, check whether dump files and abnormal operator compilation information (.o and .json files) exist in the dfx/data-dump directory, and whether log files exist in the dfx/log/host/cann directory. If the files do not exist, the AI Core error information cannot be extracted using the msaicerr tool.