iCache Data Verification Fault

Symptom

The event logs on the device (slog/dev-os-id/run/event/event_*.log) contain the keyword event_id=0x80C98000.

On the Atlas A2 training product/Atlas A2 inference product, the trace logs (in the $HOME/ascend/atrace/ directory by default) contain the keyword stars_print_error_pc_icache_and_hbm_info.

2024-04-22-09-06-17/hisi_logs/device-2/20240422090623-533810000/log/ts.log:5177:[ERROR] TSCH(-1,null):2024-04-20-17:02:52.772.875 35906 (dieid:0,cpuid:0) aicore.c:767 stars_print_error_pc_icache_and_hbm_info: stat for dump pc start, aiv_id=47, icache_miss_num=8161, hbm_miss_num=0, compare_num=32, compare_fail_num=0

On the Atlas inference product, the trace logs (in the $HOME/ascend/atrace/ directory by default) contain the keyword check_error_pc_icache_and_hbm_info.

[ERROR] TSCH(-1,null):2024-09-04-00:12:52.986.322 438 (dieid:0,cpuid:0) aicore_icache_plat.c:848 check_error_pc_icache_and_hbm_info: stat for dump pc start, aic_id=1, icache_miss_num=8176, hbm_miss_num=0, compare_num=17, compare_fail_num=0

Fault Root Causes

Locate the error keyword and check the value of compare_fail_num. If the value is not 0, the iCache memory bit error occurs.

Solution

Click Health Management Fault Definition to obtain the manual of the corresponding version. The description of the iCache memory bit error fault is as follows (some key fields are listed):

Event ID	0x80C98000
Event Name	The AI Core instruction data fails to be verified.
Fault Description/Possible Cause	The iCache data is inconsistent with the GM data. The possible causes are as follows: iCache data changes. The GM data is modified.
Impact	The current AI task fails. If the AI Core is not restored, subsequent AI tasks also fail.
System Action	TSFW reports faults to the fault management module through TSDrv. TSFW records error logs. TSFW returns a task failure message through the service plane. TSFW resets the AIC. If the AIC is successfully reset, TSFW reports through TSDrv to clear the fault events. If the AIC fails to be reset, the core is removed (after the core is isolated, it will no longer be used for service scheduling) and an error log is recorded.
System Handling Suggestion	Exit the AI training task and execute it again or initiate an inference request again. If the AI task fails to be executed again, you are advised to reset the SoC. If the fault persists, you are advised to return the device to the factory for repair.

Parent topic: Typical Cases