AI Core Timeout Fault
Symptom
The following error information is displayed in the host application log file (log/[run|debug]/plog/plog-pid _*.log) :
[ERROR] RUNTIME(3813,python3):2025-01-10-17:53:52.846.403 [device_error_proc.cc:1409] 3813 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999 [ERROR] RUNTIME(3813,python3):2025-01-10-17:53:52.846.427 [device_error_proc.cc:1409] 3813 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dield:0), serial number is 67, there is an aivector error exception, core id is 18, error code = 0, dump info: pc start: 0x124800000000, current: 0x12480000008c, vec error info: 0x5a1e95250c, mte error info: 0xf3ff16bb18, ifu error info: 0xcd34130e0080, ccu error info : 0x8405bf692593f308, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c1c001c00. [ERROR] RUNTIME(3813,python3):2025-01-10-17:53:52.846.526 [device_error_proc.cc:1421] 3813 ProcessStarsCoreErrorInfo: report error module_type=5, module_name=EZ9999 [ERROR] RUNTIME(3813,python3):2025-01-10-17:53:52.846.528 [device_error_proc.cc:1421] 3813 ProcessStarsCoreErrorInfo: The extend info: errcode:(0, 0, 0) errorStr: timeout or trap error. fixp_error0 info: 0xf16bb18, fixp_error1 info: 0xf3 fsmId:0, tslot:5, thread:0, ctxid:0, blk:11, subblk:0, subErrType:4.
The event logs on the device (slog/dev-os-id/run/event/event_*.log) contain the keyword event_id=0x80C98001 or event_id=0x80CB8001.
Fault root causes
Use the ascend-dmi tool to perform a stress test on the AI Core. If the stress test is abnormal and the message "a timeout error occurred" is displayed, the AI Core times out. The following is an example of the error message:
Hardware:
aicore:
FAIL
*** Some processes are seizing the NPU. Test results may be affected.
*** Device 4: a timeout error occurred.
The ascend-dmi tool needs to be installed separately. The following is an example of the command for performing a pressure test on the AI Core:
ascend-dmi --dg -i aicore -s -q
The ascend-dmi tool is contained in the MindCluster ToolBox software package. To query the version mapping between the software and CANN, click Link. For details about how to install and use the ascend-dmi tool, see Link.