驱动：24.1.rc2
固件：7.3.0.1.231
使用镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts
已按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-14B 中的操作步骤，完成了：
1、稀疏量化、量化权重切分及压缩
python3 quant_qwen.py --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B/ --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S/ --calib_file ../common/cn_en.jsonl --w_bit 4 --a_bit 8 --fraction 0.011 --co_sparse True --device_type npu --use_sigma True --is_lowbit True --sigma_factor 4.0 --anti_method m4
torchrun --nproc_per_node 2 -m examples.convert.model_slim.sparse_compressor --multiprocess_num 4 --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC
2、对话测试
torchrun --nproc_per_node 2 --master_port 20037 -m examples.run_pa --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC --max_output_length 20
上述操作都正常，但是进行服务化推理的时候报错：
[2025-03-04 16:16:34,884] [340] [281464127078752] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,942] [338] [281464018104672] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,951] [338] [281464018104672] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,952] [338] [281464018104672] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281472827437616 id of vcache is 281472827437520
2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[338]:
EXCEPTION STREAM:
Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[338]:
EXCEPTION STREAM:
Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-04 16:16:42,603 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[1] PID[340]:
EXCEPTION STREAM:
Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[1] PID[340]:
EXCEPTION STREAM:
Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-04 16:16:42,604 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
点赞
收藏
回复
分享
举报
已解决
驱动：24.1.rc2
固件：7.3.0.1.231
使用镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts
已按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-14B 中的操作步骤，完成了：
1、稀疏量化、量化权重切分及压缩
python3 quant_qwen.py --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B/ --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S/ --calib_file ../common/cn_en.jsonl --w_bit 4 --a_bit 8 --fraction 0.011 --co_sparse True --device_type npu --use_sigma True --is_lowbit True --sigma_factor 4.0 --anti_method m4
torchrun --nproc_per_node 2 -m examples.convert.model_slim.sparse_compressor --multiprocess_num 4 --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC
2、对话测试
torchrun --nproc_per_node 2 --master_port 20037 -m examples.run_pa --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC --max_output_length 20
上述操作都正常，但是进行服务化推理的时候报错：
[2025-03-04 16:16:34,884] [340] [281464127078752] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,942] [338] [281464018104672] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,951] [338] [281464018104672] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-03-04 16:16:34,952] [338] [281464018104672] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281472827437616 id of vcache is 281472827437520
2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[338]:
EXCEPTION STREAM:
Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[338]:
EXCEPTION STREAM:
Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-04 16:16:42,603 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[1] PID[340]:
EXCEPTION STREAM:
Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[1] PID[340]:
EXCEPTION STREAM:
Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2
Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-04 16:16:42,604 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...