环境
硬件：Ascend 310（四张Atlas 300I Duo）
系统环境：openEuler 22.03 (LTS-SP4)
驱动版本：npu-smi 24.1.0.1 Version: 24.1.0.1
CANN版本：
按照https://gitee.com/ascend/msit/tree/master/msmodelslim的流程安装了msModelSlim，也安装了中间报错缺少的一系列包
控制版本transformers==4.43.1
出现问题
执行脚本进行模型量化：
报错
error code is 507013
EH9999: Inner Error!
（如果有code自查和解决方案的话我可以摸索一下
5、请附上您出现问题页面的整屏截图或者日志信息;
feature process: 0%| | 0/5 [00:01<?, ?it/s]Traceback (most recent call last): File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 457, in rollback_names_process self.act_states = get_features(model, calib_data[:5], "features.npy", enable_tensor_dump) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "quant_funcs.py", line 1417, in quant_funcs.get_features File "quant_funcs.py", line 1418, in quant_funcs.get_features File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 175, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1054, in forward outputs = self.model( ^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 856, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/accelerate_adapter/hook_adapter.py", line 233, in pre_forward args, kwargs = self.old_hook.pre_forward(module, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 369, in pre_forward return send_to_device(args, self.execution_device), send_to_device( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 169, in send_to_device return honor_type( ^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 81, in honor_type return type(obj)(generator) ^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 170, in tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 153, in send_to_device return tensor.to(device, non_blocking=non_blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013 [ERROR] 2025-06-20-13:49:22 (PID:2419492, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EI9999: Inner Error! The error from device(1), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000080880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00801004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:811] EI9999: [PID: 2419492] 2025-06-20-13:49:22.955.604 Memory async copy failed, device_id=1, stream_id=3, task_id=1491, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=3164160[FUNC:GetError][FILE:stream.cc][LINE:1124] TraceBack (most recent call last): rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] synchronize stream failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[2419492]: EXCEPTION STREAM: Exception info:TGID=2419492, model id=65535, stream id=3, stream phase=SCHEDULE Message info[0]:RTS_HWTS: hwts sdma error, slot_id=15, stream_id=3 Other info[0]:time=2025-06-20-13:49:22.216.518, function=int_process_hwts_sdma_error, line=1376, error code=0x20b The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/data/opt/llm_model/examples/convert/model_slim/quantifier.py", line 172, in quantifier.convert(tokenized_calib_data, save_directory, args.disable_level, part_file_size=args.part_file_size) File "/data/opt/llm_model/examples/convert/model_slim/quantifier.py", line 116, in convert calibrator = Calibrator(self.model, self.quant_config, calib_data=tokenized_data, disable_level=disable_level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 164, in init self.rollback_names_process(model) File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 459, in rollback_names_process raise Exception("Please check the model and calibration data, " Exception: ('Please check the model and calibration data, ensure that your model can run with model((calib_data[0])).', RuntimeError("copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013\n[ERROR] 2025-06-20-13:49:22 (PID:2419492, Device:1, RankID:-1) ERR00100 PTA call acl api failed\n[Error]: System Direct Memory Access (DMA) hardware execution error. \n Rectify the fault based on the error information in the ascend log.\nEI9999: Inner Error!\n The error from device(1), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000080880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00801004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:811]\nEI9999: [PID: 2419492] 2025-06-20-13:49:22.955.604 Memory async copy failed, device_id=1, stream_id=3, task_id=1491, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=3164160[FUNC:GetError][FILE:stream.cc][LINE:1124]\n TraceBack (most recent call last):\n rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]\n synchronize stream failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]\n\n\nDEVICE[1] PID[2419492]: \nEXCEPTION STREAM:\n Exception info:TGID=2419492, model id=65535, stream id=3, stream phase=SCHEDULE\n Message info[0]:RTS_HWTS: hwts sdma error, slot_id=15, stream_id=3\n Other info[0]:time=2025-06-20-13:49:22.216.518, function=int_process_hwts_sdma_error, line=1376, error code=0x20b")) [W620 13:49:22.567779665 compiler_depend.ts:526] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.967.799 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeUsedDevices) [W620 13:49:22.569388304 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.969.476 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:22.570896391 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.970.973 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.605944470 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.006.079 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.607438856 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.007.611 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.627593824 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.027.742 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.629005619 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.029.179 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.630418563 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.030.583 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.631845857 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.032.016 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.633238981 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.033.416 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.634642105 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.034.819 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.636042449 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.036.216 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.637443283 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.037.625 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.638827237 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.039.000 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.640256171 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.040.415 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.641638075 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.041.815 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.643053259 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.043.232 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache)
点赞
收藏
回复
分享
举报
已解决
环境
硬件：Ascend 310（四张Atlas 300I Duo）
系统环境：openEuler 22.03 (LTS-SP4)
驱动版本：npu-smi 24.1.0.1 Version: 24.1.0.1
CANN版本：
按照https://gitee.com/ascend/msit/tree/master/msmodelslim的流程安装了msModelSlim，也安装了中间报错缺少的一系列包
控制版本transformers==4.43.1
出现问题
执行脚本进行模型量化：
报错
error code is 507013
EH9999: Inner Error!
（如果有code自查和解决方案的话我可以摸索一下
5、请附上您出现问题页面的整屏截图或者日志信息;
feature process: 0%| | 0/5 [00:01<?, ?it/s]Traceback (most recent call last): File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 457, in rollback_names_process self.act_states = get_features(model, calib_data[:5], "features.npy", enable_tensor_dump) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "quant_funcs.py", line 1417, in quant_funcs.get_features File "quant_funcs.py", line 1418, in quant_funcs.get_features File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 175, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1054, in forward outputs = self.model( ^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 856, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/accelerate_adapter/hook_adapter.py", line 233, in pre_forward args, kwargs = self.old_hook.pre_forward(module, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/hooks.py", line 369, in pre_forward return send_to_device(args, self.execution_device), send_to_device( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 169, in send_to_device return honor_type( ^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 81, in honor_type return type(obj)(generator) ^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 170, in tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/accelerate/utils/operations.py", line 153, in send_to_device return tensor.to(device, non_blocking=non_blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013 [ERROR] 2025-06-20-13:49:22 (PID:2419492, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EI9999: Inner Error! The error from device(1), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000080880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00801004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:811] EI9999: [PID: 2419492] 2025-06-20-13:49:22.955.604 Memory async copy failed, device_id=1, stream_id=3, task_id=1491, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=3164160[FUNC:GetError][FILE:stream.cc][LINE:1124] TraceBack (most recent call last): rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] synchronize stream failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[2419492]: EXCEPTION STREAM: Exception info:TGID=2419492, model id=65535, stream id=3, stream phase=SCHEDULE Message info[0]:RTS_HWTS: hwts sdma error, slot_id=15, stream_id=3 Other info[0]:time=2025-06-20-13:49:22.216.518, function=int_process_hwts_sdma_error, line=1376, error code=0x20b The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/data/opt/llm_model/examples/convert/model_slim/quantifier.py", line 172, in quantifier.convert(tokenized_calib_data, save_directory, args.disable_level, part_file_size=args.part_file_size) File "/data/opt/llm_model/examples/convert/model_slim/quantifier.py", line 116, in convert calibrator = Calibrator(self.model, self.quant_config, calib_data=tokenized_data, disable_level=disable_level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 164, in init self.rollback_names_process(model) File "/data/root/anaconda/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 459, in rollback_names_process raise Exception("Please check the model and calibration data, " Exception: ('Please check the model and calibration data, ensure that your model can run with model((calib_data[0])).', RuntimeError("copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013\n[ERROR] 2025-06-20-13:49:22 (PID:2419492, Device:1, RankID:-1) ERR00100 PTA call acl api failed\n[Error]: System Direct Memory Access (DMA) hardware execution error. \n Rectify the fault based on the error information in the ascend log.\nEI9999: Inner Error!\n The error from device(1), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000080880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00801004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:811]\nEI9999: [PID: 2419492] 2025-06-20-13:49:22.955.604 Memory async copy failed, device_id=1, stream_id=3, task_id=1491, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=3164160[FUNC:GetError][FILE:stream.cc][LINE:1124]\n TraceBack (most recent call last):\n rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]\n synchronize stream failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]\n\n\nDEVICE[1] PID[2419492]: \nEXCEPTION STREAM:\n Exception info:TGID=2419492, model id=65535, stream id=3, stream phase=SCHEDULE\n Message info[0]:RTS_HWTS: hwts sdma error, slot_id=15, stream_id=3\n Other info[0]:time=2025-06-20-13:49:22.216.518, function=int_process_hwts_sdma_error, line=1376, error code=0x20b")) [W620 13:49:22.567779665 compiler_depend.ts:526] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.967.799 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeUsedDevices) [W620 13:49:22.569388304 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.969.476 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:22.570896391 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:22.970.973 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.605944470 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.006.079 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.607438856 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.007.611 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.627593824 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.027.742 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.629005619 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.029.179 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.630418563 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.030.583 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.631845857 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.032.016 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.633238981 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.033.416 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.634642105 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.034.819 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.636042449 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.036.216 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.637443283 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.037.625 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.638827237 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.039.000 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.640256171 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.040.415 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache) [W620 13:49:23.641638075 compiler_depend.ts:508] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.041.815 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W620 13:49:23.643053259 compiler_depend.ts:151] Warning: NPU warning, error code is 507013[Error]: [Error]: System Direct Memory Access (DMA) hardware execution error. Rectify the fault based on the error information in the ascend log. EH9999: Inner Error! rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]EH9999: [PID: 2419492] 2025-06-20-13:49:23.043.232 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function empty_cache)