
希望使用MMad接口而不使用Matmul高阶api
Kernel函数代码来源于官方示例代码
编译代码参考samples/operator/AddKernel/AddKernelInvocation中编译代码
请问报错的原因是什么,如何解决?
报错结果:
[EVENT] PROFILING(2303270,matmul_custom_npu):2024-02-21-10:28:21.873.646 [prof_atls_plugin.cpp:154] >>> (tid:2303270) Module[8] register callback of ctrl handle.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.873.856 [engine.cc:3775]2303270 ProcLogicCqReport:Task run failed, device_id=0, stream_id=3, task_id=0, sqe_type=0(ffts), errType=0x1(task exception), sqSwStatus=0
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.107 [runtime.cc:4347]2303270 SetWatchDogDevStatus:There is errInfo of devId=0, tsId=0
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.127 [device_error_proc.cc:1138]2303270 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.134 [device_error_proc.cc:1138]2303270 ProcessStarsCoreErrorInfo:The error from device(chipId:3, dieId:0), serial number is 29, there is an aivec error exception, core id is 20, error code = 0, dump info: pc start: 0x1240c0019000, current: 0x1240c0019644, vec error info: 0xe100a5c730, mte error info: 0xef3dabf0f6, ifu error info: 0x10661590f3f00, ccu error info: 0x96d8474328000b6, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100340000.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.164 [device_error_proc.cc:1150]2303270 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.171 [device_error_proc.cc:1150]2303270 ProcessStarsCoreErrorInfo:The extend info: errcode:(0, 0x200000000000000, 0) errorStr: The MPU address access is invalid. fixp_error0 info: 0xdabf0f6, fixp_error1 info: 0xef fsmId:0, tslot:0, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.188 [task_info.cc:1714]2303270 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.195 [task_info.cc:1612]2303270 PreCheckTaskErr:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.201 [task_info.cc:1612]2303270 PreCheckTaskErr:Kernel task happen error, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.255 [task_info.cc:1574]2303270 PrintErrorInfoForDavinciTask:Aicore kernel execute failed, device_id=0, stream_id=3, report_stream_id=3, task_id=0, flip_num=0, fault kernel_name=matmul_custom, program id=0, hash=8935204169602055009.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.281 [task_info.cc:1516]2303270 GetArgsInfo:[AIC_INFO] args(0 to 3) after execute:0x1240c0015000, 0x1240c0016000, 0x1240c0017000,
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.288 [task_info.cc:1519]2303270 GetArgsInfo:print 1 Times totalLen=(3*8)Bytes, argsSize=24
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.296 [task_info.cc:1578]2303270 PrintErrorInfoForDavinciTask:[AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.326 [stream.cc:1480]2303270 GetError:Stream Synchronize failed, stream_id=3, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.333 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.337 [stream.cc:1483]2303270 GetError:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.347 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.353 [stream.cc:1483]2303270 GetError:Aicore kernel execute failed, device_id=0, stream_id=3, report_stream_id=3, task_id=0, flip_num=0, fault kernel_name=matmul_custom, program id=0, hash=8935204169602055009.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.362 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.368 [stream.cc:1483]2303270 GetError:[AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.376 [logger.cc:472]2303270 StreamSynchronize:Stream synchronize failed, stream_id=3
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.388 [api_c.cc:787]2303270 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.393 [error_message_manage.cc:50]2303270 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.401 [error_message_manage.cc:50]2303270 FuncErrorReason:rtStreamSynchronize execute failed, reason=[vector core exception]
[ERROR] ASCENDCL(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.412 [stream.cpp:121]2303270 aclrtSynchronizeStream: synchronize stream failed, runtime result = 507035
main.cpp:67 aclError:507035
希望使用MMad接口而不使用Matmul高阶api
Kernel函数代码来源于官方示例代码
编译代码参考samples/operator/AddKernel/AddKernelInvocation中编译代码
请问报错的原因是什么,如何解决?
报错结果:
[EVENT] PROFILING(2303270,matmul_custom_npu):2024-02-21-10:28:21.873.646 [prof_atls_plugin.cpp:154] >>> (tid:2303270) Module[8] register callback of ctrl handle.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.873.856 [engine.cc:3775]2303270 ProcLogicCqReport:Task run failed, device_id=0, stream_id=3, task_id=0, sqe_type=0(ffts), errType=0x1(task exception), sqSwStatus=0
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.107 [runtime.cc:4347]2303270 SetWatchDogDevStatus:There is errInfo of devId=0, tsId=0
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.127 [device_error_proc.cc:1138]2303270 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.134 [device_error_proc.cc:1138]2303270 ProcessStarsCoreErrorInfo:The error from device(chipId:3, dieId:0), serial number is 29, there is an aivec error exception, core id is 20, error code = 0, dump info: pc start: 0x1240c0019000, current: 0x1240c0019644, vec error info: 0xe100a5c730, mte error info: 0xef3dabf0f6, ifu error info: 0x10661590f3f00, ccu error info: 0x96d8474328000b6, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100340000.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.164 [device_error_proc.cc:1150]2303270 ProcessStarsCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.171 [device_error_proc.cc:1150]2303270 ProcessStarsCoreErrorInfo:The extend info: errcode:(0, 0x200000000000000, 0) errorStr: The MPU address access is invalid. fixp_error0 info: 0xdabf0f6, fixp_error1 info: 0xef fsmId:0, tslot:0, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.188 [task_info.cc:1714]2303270 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.195 [task_info.cc:1612]2303270 PreCheckTaskErr:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.201 [task_info.cc:1612]2303270 PreCheckTaskErr:Kernel task happen error, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.255 [task_info.cc:1574]2303270 PrintErrorInfoForDavinciTask:Aicore kernel execute failed, device_id=0, stream_id=3, report_stream_id=3, task_id=0, flip_num=0, fault kernel_name=matmul_custom, program id=0, hash=8935204169602055009.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.281 [task_info.cc:1516]2303270 GetArgsInfo:[AIC_INFO] args(0 to 3) after execute:0x1240c0015000, 0x1240c0016000, 0x1240c0017000,
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.288 [task_info.cc:1519]2303270 GetArgsInfo:print 1 Times totalLen=(3*8)Bytes, argsSize=24
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.296 [task_info.cc:1578]2303270 PrintErrorInfoForDavinciTask:[AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.326 [stream.cc:1480]2303270 GetError:Stream Synchronize failed, stream_id=3, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.333 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.337 [stream.cc:1483]2303270 GetError:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.347 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.353 [stream.cc:1483]2303270 GetError:Aicore kernel execute failed, device_id=0, stream_id=3, report_stream_id=3, task_id=0, flip_num=0, fault kernel_name=matmul_custom, program id=0, hash=8935204169602055009.
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.362 [stream.cc:1483]2303270 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.368 [stream.cc:1483]2303270 GetError:[AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.376 [logger.cc:472]2303270 StreamSynchronize:Stream synchronize failed, stream_id=3
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.388 [api_c.cc:787]2303270 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.393 [error_message_manage.cc:50]2303270 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.401 [error_message_manage.cc:50]2303270 FuncErrorReason:rtStreamSynchronize execute failed, reason=[vector core exception]
[ERROR] ASCENDCL(2303270,matmul_custom_npu):2024-02-21-10:28:21.889.412 [stream.cpp:121]2303270 aclrtSynchronizeStream: synchronize stream failed, runtime result = 507035
main.cpp:67 aclError:507035