Using the msaicerr Tool to Analyze AI Core Errors

Procedure

1. Log in to the host server as the running user.
2. Use the msaicerr tool to quickly locate the key causes of AI Core errors.

   Go to the ${Toolkit installation path}/tools/msaicerr directory and run the following command to extract key information related to the AI Core error based on the information collected in Collecting AI Core Error Information. In the following command, aic_err_info_timestamp indicates the directory for storing AI Core error information. Replace it with the actual directory.

   python3 msaicerr.py -p ${HOME}/aic_err_info_timestamp -out $HOME/result

   In the preceding command example, the -p parameter is used to specify the directory for storing fault information, for example, ${HOME}/aic_err_info_timestamp. The -out parameter is used to specify the path for storing the parsing result file, for example, $HOME/result; if the path is not specified, the parsing result is stored in the current path where the command is executed by default.

   Note: Do not execute the msaicerr tool in the directory specified by the -p parameter or its subdirectory. For example, you cannot execute the msaicerr tool in the aic_err_info_timestamp directory or its subdirectory. The directory specified by the -out parameter must not be the directory specified by the -p parameter or its subdirectory. Otherwise, the tool parsing may be suspended or fail.

   After the command is executed, you can analyze and locate the fault based on the prompt information in the info.txt file, the path of which is displayed on the terminal interface. Pay attention to the key information described in the following table. If the information collected in Collecting AI Core Error Information contains multiple AI Core errors, the msaicerr tool parses the AI Core error that occurs for the first time based on the log time.

   Table 1 Key Information

   Key Information	Possible Cause	Typical Cases and Handling Methods
   Failed to execute the built-in sample operator. Check the environment.	The operating environment is abnormal.	System Environment/Hardware Problems
   Failed to execute the single-operator test case. The operator logic may be incorrect.	Errors occur in single-operator implementation or compilation process. Check whether the operator is a user custom operator or a built-in CANN operator based on the displayed information. Check the debug_info.txt file. If the following information is displayed, the tool automatically generates a single-operator case script. You can run the script to reproduce the single-operator problem and rectify the fault as prompted. If the problem cannot be reproduced or rectified, contact technical support. After obtaining the logs, click here to contact technical support. Run 'export PYTHONPATH=/usr/local/Ascend/CANN-7.3/tools/msaicerr/:$PYTHONPATH;cd /usr/local/Ascend/CANN-7.3/tools/msaicerr;python3 /home/xxxxxxx/xxx/info_xxxx/aicerror_xxxx/test_single_op.py' can test op!	Single-operator Running Error
   Atomic add has a precision overflow. Check the operator precision. Note that if tasks are concurrently executed on the NPU, a false warning may be reported.	Overflow occurs due to precision problems.	Accuracy Overflow/Underflow During Atomic Add
   The input/output memory address of the operator is abnormal (or the original dumped data fails). Check the framework or application.	The input and output data addresses of the operator are abnormal. The memory allocated to the framework is incorrect. In this case, check whether the framework is GE or other frameworks and contact technical support. After obtaining the logs, click here to contact technical support.	Abnormal Operator Input/Output Data Addresses Dump Data Failed
   If the arguments are inconsistent before and after operator execution, memory access may be out of bounds. You are advised to use the memory error detection model to locate the fault.	The input and output parameters of the operator are abnormal.	Inconsistent Dispensation Before And After The Operator Input Arg
   The number of AI Cores in the environment is less than that required by the operator.	The number of AI Cores in the environment is less than that required by the operator.	Check whether the number of AI Cores in the environment where the msaicerr tool is used is the same as that in the actual environment. If yes, contact technical support. After obtaining the logs, click here to contact technical support.
   The memset or atomic_clean operator is not inserted before this operator in the graph, while memory cleanup is required before operator execution.	Graph build is abnormal.	Contact technical support. After obtaining the logs, click here to contact technical support.
   The set_flag and wait_flag instructions are not used together in the operator code.	The set_flag and wait_flag instructions do not match.	For CANN built-in operators, contact technical support. After obtaining the logs, click here to contact technical support. For custom operators, check the operator code by yourself.
   The single-operator test case is successfully executed. In case of an unknown error mode, you are advised to: (1) check the operator again by using the msSanitizer tool. (2) If out-of-bounds memory access occurs on other operators, you are advised to enable memory error detection with op_debug_config=oom and then check the operators. For details: https://www.hiascend.com/zh/document. (3) For details about the framework, contact technical support.	Single-operator reproduction is successful.	Contact technical support. After obtaining the logs, click here to contact technical support.
   Internal error. Contact technical support.	Internal error.	Contact technical support. After obtaining the logs, click here to contact technical support.
   The maintenance and test information is insufficient or the format is incorrect, contact technical support.	The maintenance and test information is insufficient or the format is incorrect. Modify the maintenance and test data or format based on the specific error information.	Contact technical support. After obtaining the logs, click here to contact technical support.

   If the msaicerr tool fails to be executed:

   1. Check whether the prerequisites for using the tool are met and whether the information collected in Collecting AI Core Error Information is complete.
   2. Check the operator parameters by referring to Operator Input Argument Errors.
   3. If the fault cannot be located, After obtaining the logs, click here to contact technical support.