aicore error
收藏回复举报
aicore error
t('forum.solved') 已解决
发表于2024-05-23 10:27:58
0 查看
算子测试的时候发现算子输出全是0.打开export ASCEND_SLOG_PRINT_TO_STDOUT=1 

 export ASCEND_LAUNCH_BLOCKING=1 后log输出如下:
[INFO] ATRACE(2235002,python):2024-05-23-02:11:48.308.531 [trace_attr.c:164](tid:2235002) attr init success, timeout=0, driver version=467731. 

[INFO] ATRACE(2235002,python):2024-05-23-02:11:48.308.620 [trace_recorder.c:114](tid:2235002) use root path: /data1/home/houyazhou/ascend 

[EVENT] PROFILING(2235002,python):2024-05-23-02:11:48.438.588 [msprof_callback_impl.cpp:334] >>> (tid:2235002) Started to register profiling ctrl callback. 

[EVENT] PROFILING(2235002,python):2024-05-23-02:11:48.438.786 [msprof_callback_impl.cpp:341] >>> (tid:2235002) Started to register profiling hash id callback. 

[INFO] PROFILING(2235002,python):2024-05-23-02:11:48.438.802 [prof_atls_plugin.cpp:83] >>> (tid:2235002) RegisterProfileCallback, callback type is 7 

[EVENT] PROFILING(2235002,python):2024-05-23-02:11:48.438.810 [msprof_callback_impl.cpp:348] >>> (tid:2235002) Started to register profiling enable host freq callback. 

[INFO] PROFILING(2235002,python):2024-05-23-02:11:48.438.816 [prof_atls_plugin.cpp:83] >>> (tid:2235002) RegisterProfileCallback, callback type is 8 

[INFO] RUNTIME(2235002,python):2024-05-23-02:11:48.445.349 [runtime.cc:5091] 2235002 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set 

[INFO] PROFILING(2235002,python):2024-05-23-02:11:48.447.789 [prof_atls_plugin.cpp:160] >>> (tid:2235002) Module[7] register callback of ctrl handle. 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.460.647 [client_manager.cpp:396][IsHostEnvironment][tid:2235002] [TsdClient] logicDeviceId is [0], hostaicpunum[0] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.460.672 [client_manager.cpp:425][GetClientRunMode][tid:2235002] runningMode:0 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.460.680 [client_manager.cpp:104][GetInstance][tid:2235002] curmode:1 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.460.783 [process_mode_manager.cpp:125][OpenProcess][tid:2235002] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.366 [process_mode_manager.cpp:409][InitTsdClient][tid:2235002] [TsdClient] deviceId[0] begin to init hdc client 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.493 [version_verify.cpp:34][SetVersionInfo][tid:2235002] VersionVerify: send client version to server 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.521 [version_verify.cpp:50][SetVersionInfo][tid:2235002] send feature_info:{msg_type:35, features:{check before send aicpu package,}} 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.529 [version_verify.cpp:50][SetVersionInfo][tid:2235002] send feature_info:{msg_type:37, features:{check before send open qs message,}} 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.839 [version_verify.cpp:66][PeerVersionCheck][tid:2235002] VersionVerify: Check client version info, server[1230], client[1230] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.852 [version_verify.cpp:87][ParseVersionInfo][tid:2235002] VersionVerify: pass client version info success 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.859 [hdc_client.cpp:276][CheckHdcConnection][tid:2235002] Service[2] create hdc success 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.879 [version_verify.cpp:120][SpecialFeatureCheck][tid:2235002] VersionVerify: new type[35], supported 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.461.951 [process_mode_manager.cpp:811][GetDeviceCheckCode][tid:2235002] [TsdClient][deviceId=0] [sessionId=1] wait package info respond 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.103 [process_mode_manager.cpp:1905][SendExtendPackage][tid:2235002] device not support extend pkg skip send 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.116 [process_mode_manager.cpp:409][InitTsdClient][tid:2235002] [TsdClient] deviceId[0] begin to init hdc client 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.193 [version_verify.cpp:34][SetVersionInfo][tid:2235002] VersionVerify: send client version to server 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.204 [version_verify.cpp:50][SetVersionInfo][tid:2235002] send feature_info:{msg_type:35, features:{check before send aicpu package,}} 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.211 [version_verify.cpp:50][SetVersionInfo][tid:2235002] send feature_info:{msg_type:37, features:{check before send open qs message,}} 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.432 [version_verify.cpp:66][PeerVersionCheck][tid:2235002] VersionVerify: Check client version info, server[1230], client[1230] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.442 [version_verify.cpp:87][ParseVersionInfo][tid:2235002] VersionVerify: pass client version info success 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.448 [hdc_client.cpp:276][CheckHdcConnection][tid:2235002] Service[2] create hdc success 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.457 [process_mode_manager.cpp:456][ConstructOpenMsg][tid:2235002] [TsdClient] tsd get process sign successfully, procpid[2235002] signSize[48] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.464 [version_verify.cpp:112][SpecialFeatureCheck][tid:2235002] VersionVerify: previous type[6], supported 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.462.482 [process_mode_manager.cpp:142][OpenProcess][tid:2235002] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.624.961 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:2235002] [TsdClient] it is unnecessary of current mode[0] chiptype[5] to grant queue auth to aicpusd 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.624.987 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:2235002] enter into OpenInHost deviceid[0] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.625.013 [client_manager.cpp:396][IsHostEnvironment][tid:2235002] [TsdClient] logicDeviceId is [0], hostaicpunum[0] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.625.022 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:2235002] host cpu not support 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.625.031 [process_mode_manager.cpp:172][OpenProcess][tid:2235002] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success 

[INFO] RUNTIME(2235002,python):2024-05-23-02:11:48.630.631 [device.cc:336] 2235002 Init: isAddrFlat:0 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.631.865 [tsd_client.cpp:182][TsdCapabilityGet][tid:2235002] TsdCapabilityGet Begin. 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.631.889 [process_mode_manager.cpp:1117][CapabilityGet][tid:2235002] [ProcessModeManager] enter into CapabilityGet process deviceId[0] type[0] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.631.902 [version_verify.cpp:112][SpecialFeatureCheck][tid:2235002] VersionVerify: previous type[38], supported 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.631.935 [process_mode_manager.cpp:1140][CapabilityGet][tid:2235002] [TsdClient][deviceId=0] [sessionId=1] [type=0]send capability success  

[INFO] TDT(2235002,python):2024-05-23-02:11:48.632.100 [process_mode_manager.cpp:682][PidQosMsgProc][tid:2235002] [TsdClient] PidQosMsgProc recvMsg realDeviceId[0] msgType[39] localDevId[0] rspCode[0],pidqos[1] 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.632.112 [process_mode_manager.cpp:1145][CapabilityGet][tid:2235002] [TsdClient][logicDeviceId_=0][type=0][pidQos=1][ret=0]recv capability respond finish 

[INFO] RUNTIME(2235002,python):2024-05-23-02:11:48.656.316 [npu_driver.cc:5505] 2235340 GetDeviceStatus: GetDeviceStatus status=1. 

[INFO] ATRACE(2235002,python):2024-05-23-02:11:48.657.370 [tracer_schedule.c:57](tid:2235002) create object RUNTIME_ATRACE_DEV0_TS0 success, exitSave(false). 

[INFO] TDT(2235002,python):2024-05-23-02:11:48.657.400 [client_manager.cpp:163][SetProfilingCallback][tid:2235002] [TsdClient] set profiling callback success 

MatrixBCols = 1 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.660.446 [stars_engine.cc:1267]2235340 ProcLogicCqReport:Task run failed, device_id=0, stream_id=7, task_id=4, sqe_type=0(ffts), errType=0x1(task exception), sqSwStatus=0 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.664.789 [device_error_proc.cc:1170]2235340 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.664.888 [device_error_proc.cc:1170]2235340 ProcessStarsCoreErrorInfo:The error from device(chipId:0, dieId:0), serial number is 275, there is an aicore error exception, core id is 19, error code = 0x10, dump info: pc start: 0x1240c002508c, current: 0x1240c002657c, vec error info: 0, mte error info: 0xa8f5462cb8, ifu error info: 0x40e121e352bc0, ccu error info: 0x804000004c800058, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100240000. 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.679.578 [device_error_proc.cc:1182]2235340 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.679.597 [device_error_proc.cc:1182]2235340 ProcessStarsCoreErrorInfo:The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x5462cb8, fixp_error1 info: 0xa8 fsmId:1, tslot:5, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4. 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.679.821 [task_info.cc:1864]2235340 SetStarsResultForDavinciTask:AICORE Kernel task happen error, retCode=0x26. 

[INFO] RUNTIME(2235002,python):2024-05-23-02:11:49.679.836 [runtime.cc:5494] 2235340 SetWatchDogDevStatus: There is errInfo of devId=0, tsId=0 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.679.858 [task_info.cc:1754]2235340 PreCheckTaskErr:report error module_type=5, module_name=EZ9999 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.679.866 [task_info.cc:1754]2235340 PreCheckTaskErr:Kernel task happen error, retCode=0x26, [aicore exception]. 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.680.155 [task_info.cc:1678]2235340 PrintErrorInfoForDavinciTask:Aicore kernel execute failed, device_id=0, stream_id=7, report_stream_id=7, task_id=4, flip_num=0, fault kernel_name=_Z19matmul_custom_floatPhS_S_jjjjbb, fault kernel info ext=_Z19matmul_custom_floatPhS_S_jjjjbb, program id=0, hash=5245107584082966150. 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.680.230 [task_info.cc:1617]2235340 GetArgsInfo:[AIC_INFO] args(0 to 5) after execute:0x124042200000, 0x124042200200, 0x124042200400, 0x100000001, 0x100000001,   

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.680.242 [task_info.cc:1620]2235340 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(5*8)Bytes, argsSize=45, blockDim=1 

[ERROR] RUNTIME(2235002,python):2024-05-23-02:11:49.680.255 [task_info.cc:1682]2235340 PrintErrorInfoForDavinciTask:[AIC_INFO] after execute:args print end 

F

看到errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses。貌似是ub地址未对齐,请问可能的原因是什么?

我要发帖子