AI Core Error Information Parsing
Description
During service execution, if the log file or information printed on the screen contains AI Core error information (for example, "there is an aivec error exception" or "there is an aicore error exception"), use the AI Core error information parsing function to quickly locate the cause of the AI Core error, thus improving troubleshooting efficiency.
Applicability
Product |
Supported |
|---|---|
Atlas 350 Accelerator Card |
√ |
√ |
|
√ |
|
√ |
|
√ |
|
√ |
Notes
- To ensure the accuracy of the parsed data, clear the logs before reproducing the AI Core error.
- To avoid cyclic copy, the --output directory cannot be the --path directory or its subdirectory.
Command
# In the following command, aic_err_info_timestamp indicates the directory for storing AI Core error information. Replace it with the actual directory.
asys analyze -r=aicore_error -d=deviceId --path=${HOME}/aic_err_info_timestamp
Parameters
- r : parsing mode. This parameter is mandatory. Set it to aicore_error.
- d: This parameter is optional. It specifies the ID of the device to be operated. If this parameter is not set, the configuration of device 0 is used by default.
- path: (optional) directory for storing fault information such as logs and dump files. For details about how to collect AI Core error information, see Collecting AI Core Error Information.
If this parameter is not set, the asys tool automatically collects fault information. Automatic collection is affected by environment variables. Therefore, when you run the asys command, the environment variable values must be the same as those used during service running. Otherwise, the collected information may be incorrect. The involved environment variables are ASCEND_PROCESS_LOG_PATH, NPU_COLLECT_PATH, DUMP_GRAPH_PATH, ASCEND_WORK_PATH, ASCEND_CACHE_PATH and ASCEND_CUSTOM_OPP_PATH. For details about the environment variables and their restrictions, see Environment Variables. If these environment variables do not exist, fault information is collected from the current directory where the asys command is executed.
- output: optional parameter. Its value is used as the output directory of the asys tool. If the command does not contain the output parameter, the output is stored in the command execution directory. If the value of output is empty or invalid, the specified directory does not have the write permission, or the directory fails to be created, the asys tool exits and reports an error.
Usage Example and Output Description
# In the following command, aic_err_info_timestamp indicates the directory for storing AI Core error information. Replace it with the actual directory.
asys analyze -r=aicore_error --path=${HOME}/aic_err_info_timestamp
After the command is executed, you can analyze and locate the fault based on the path of the info.txt file displayed on the terminal interface. Pay attention to the content in Table 1. If the collected information contains multiple AI Core errors, the tool parses the AI Core error that occurs for the first time based on the log time.