一、硬件构成：主板ITX-3588J，通过PCIE连接全爱科技的QA200EP-A2推理卡（EP模式）
二、操作系统：openEuler release 22.03 LTS
三、固件驱动：24.1.RC2
四、CANN：8.0.RC3.beta1
五、MindSpore：2.4.1
目标设备为CPU，验证通过
目标设备为Ascend，验证报错
(pyms) [root@openEuler latest]# python -c "import mindspore;mindspore.set_context(device_target='Ascend');mindspore.run_check()"
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
MindSpore version: 2.4.1
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.023 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:246] TaskExceptionCallback] Run Task failed, task_id: 4, stream_id: 1, tid: 128272, device_id: 0, retcode: 507035 (Return error code unknown, ret code: 507035)
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.811 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.855 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:593] SyncStream] Sync default stream failed.
MindSpore running check failed.
Sync stream error!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ9999: Inner Error!
EZ9999: [PID: 128272] 2025-01-21-11:37:17.508.286 The error from device(chipId:0, dieId:0), serial number is 6, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x124000000100, current: 0x124000000674, vec error info: 0, mte error info: 0x73ffaffbef, ifu error info: 0xffef3ffffdfc0, ccu error info: 0xbffff7bf00000064, cube error info: 0, biu error info: 0, aic error mask: 0x65000208d000288, para base: 0x12c040569000.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1409][THREAD:128272]
TraceBack (most recent call last):
The extend info: errcode:(0, 0x200000000000000, 0) errorStr: The MPU address access is invalid. fixp_error0 info: 0xfaffbef, fixp_error1 info: 0x73 fsmId:0, tslot:1, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1421][THREAD:128272]
Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1286][THREAD:128272]
AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
Aicore kernel execute failed, device_id=0, stream_id=1, report_stream_id=1, task_id=4, flip_num=0, fault kernel_name=Mul_ee98c6628030785f610b924ab1557b31_high_performance_210000000, fault kernel info ext=none, program id=0, hash=3483618941185071924.[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
rtStreamSynchronizeWithTimeout execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53][THREAD:128272]
synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161][THREAD:128272]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_device_address.cc:284 SyncStream
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.727 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.854 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:593] SyncStream] Sync default stream failed.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.932 [mindspore/ccsrc/runtime/device/kernel_runtime_manager.cc:134] WaitTaskFinishOnDevice] SyncStream failed
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.935.260 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.935.342 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:293] SyncAllStreams] SyncStream for stream id 0 failed.
[ERROR] ME(128272,7fad2ef020,python):2025-01-21-11:37:17.935.414 [mindspore/ccsrc/runtime/hardware/device_context_manager.cc:490] WaitTaskFinishOnDevice] SyncStream failed
(pyms) [root@openEuler latest]#
想在上面硬件和系统上使用MindSpore在线推理时输出上述报错信息，想请教一下是什么原因导致Ascend侧验证失败
点赞
收藏
回复
分享
举报
已解决
一、硬件构成：主板ITX-3588J，通过PCIE连接全爱科技的QA200EP-A2推理卡（EP模式）
二、操作系统：openEuler release 22.03 LTS
三、固件驱动：24.1.RC2
四、CANN：8.0.RC3.beta1
五、MindSpore：2.4.1
目标设备为CPU，验证通过
目标设备为Ascend，验证报错
(pyms) [root@openEuler latest]# python -c "import mindspore;mindspore.set_context(device_target='Ascend');mindspore.run_check()"
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/root/miniconda3/envs/pyms/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
MindSpore version: 2.4.1
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.023 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:246] TaskExceptionCallback] Run Task failed, task_id: 4, stream_id: 1, tid: 128272, device_id: 0, retcode: 507035 (Return error code unknown, ret code: 507035)
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.811 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.510.855 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:593] SyncStream] Sync default stream failed.
MindSpore running check failed.
Sync stream error!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ9999: Inner Error!
EZ9999: [PID: 128272] 2025-01-21-11:37:17.508.286 The error from device(chipId:0, dieId:0), serial number is 6, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x124000000100, current: 0x124000000674, vec error info: 0, mte error info: 0x73ffaffbef, ifu error info: 0xffef3ffffdfc0, ccu error info: 0xbffff7bf00000064, cube error info: 0, biu error info: 0, aic error mask: 0x65000208d000288, para base: 0x12c040569000.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1409][THREAD:128272]
TraceBack (most recent call last):
The extend info: errcode:(0, 0x200000000000000, 0) errorStr: The MPU address access is invalid. fixp_error0 info: 0xfaffbef, fixp_error1 info: 0x73 fsmId:0, tslot:1, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1421][THREAD:128272]
Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1286][THREAD:128272]
AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
Aicore kernel execute failed, device_id=0, stream_id=1, report_stream_id=1, task_id=4, flip_num=0, fault kernel_name=Mul_ee98c6628030785f610b924ab1557b31_high_performance_210000000, fault kernel info ext=none, program id=0, hash=3483618941185071924.[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1084][THREAD:128272]
rtStreamSynchronizeWithTimeout execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53][THREAD:128272]
synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161][THREAD:128272]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_device_address.cc:284 SyncStream
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.727 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.854 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:593] SyncStream] Sync default stream failed.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.934.932 [mindspore/ccsrc/runtime/device/kernel_runtime_manager.cc:134] WaitTaskFinishOnDevice] SyncStream failed
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.935.260 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:276] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error.
[ERROR] DEVICE(128272,7fad2ef020,python):2025-01-21-11:37:17.935.342 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:293] SyncAllStreams] SyncStream for stream id 0 failed.
[ERROR] ME(128272,7fad2ef020,python):2025-01-21-11:37:17.935.414 [mindspore/ccsrc/runtime/hardware/device_context_manager.cc:490] WaitTaskFinishOnDevice] SyncStream failed
(pyms) [root@openEuler latest]#
想在上面硬件和系统上使用MindSpore在线推理时输出上述报错信息，想请教一下是什么原因导致Ascend侧验证失败