Locating AI Core Errors

You can perform the following steps to locate a fault. If the fault persists, contact technical support. After obtaining the logs, click here to contact technical support.

Figure 1 Locating process

In the preparation phase, you need to collect fault information, including: CANN log files, exception dump files, and operator compilation information (*.o and *.json files). For details about how to collect fault information, see Collecting AI Core Error Information.

The locating phase is as follows:

1. Locate the RAS hardware fault.

   RAS hardware faults refer to faults related to hardware reliability, availability, and serviceability.

   In the collected slog/dev-os-id /run/event/event_*.log file, find the system logs of the corresponding device at the time when the AI Core error occurs, and check whether the keyword event_id exists in the logs. If not, go to 2 to locate the fault. If yes, obtain the value of event_id (RAS hardware error code), click Health Management Fault Definition to obtain the manual of the corresponding version, and view the troubleshooting methods. For details about typical troubleshooting cases, see HBM Bit ECC Fault, iCache Data Verification Fault, and AI Core Timeout Fault.

2. Locate the NPU hardware fault.
   In the collected application logs, find the log/[run|debug]/plog/plog-pid _*.log file generated around the time when the AI Core error occurs. Check whether ECC-related errors (with the keyword "ECC" or "ECC error") exist in the log or whether multiple errors are reported on the same chip ID.

   • If no, go to 3.

     If yes, use the ascend-dmi tool to perform a pressure test on the AI Core. If the pressure test is abnormal, the hardware is faulty. In this case, contact technical support to replace the hardware. For details about typical troubleshooting cases, see AI Core Hardware Fault. If the pressure test is normal, specify another device in the program and run the program to check whether the problem recurs. If the problem recurs, go to 3 to locate the fault. If the problem does not recur, the hardware may be faulty. In this case, contact technical support to replace the hardware. After obtaining the logs, click here to contact technical support.

     The ascend-dmi tool needs to be installed separately. The following is the example command for performing a stress test on the AI Core. If the error message GENERAL_WARN or EMERGENCY_WARN is reported, the AI Core may be faulty.

     ascend-dmi --dg -i aicore -s

     The ascend-dmi tool is contained in the MindCluster ToolBox software package. To query the version mapping between the software and CANN, click Link. For details about how to install and use the ascend-dmi tool, see Link.

3. Locate the software fault.
   1. In the collected app logs, find the log/[run|debug]/plog/plog-pid _*.log file generated around the time when the AI Core error occurs. Check whether the 0x800000 error is reported. If no, go to 3.b. If yes, check the input data of the operator by referring to Index Operator Out of Range.

      Typical index operators include GatherV2, Scatter, and GatherElements.

   2. Use the msaicerr tool to analyze the information collected in the preparation phase. The msaicerr tool generates an analysis report (info.txt file). Provide the msaicerr-generated result data (including the minimum set information for analyzing AI Core errors and the analysis report) to technical support for further analysis. After obtaining the logs, click here to contact technical support.

      For details about how to use the msaicerr tool, see Using the msaicerr Tool to Analyze AI Core Errors.