(Python310) root@bms-41ba-0002:/usr/local/Ascend/atb-models/tests/modeltest# cat output_precision.log
nohup: ignoring input
INFO: Detected Ascend NPU
INFO: current model_type: pa_fp16
INFO: current test_mode: full_BoolQ
INFO: current batch_size: 1
INFO: current model_name: llama
INFO: current weight_dir: /root/model_pt/DeepSeek-R1-Distill-Llama-8B/
INFO: current model_type: pa
INFO: current data_type: fp16
INFO: current test_mode: full
INFO: current dataset: BoolQ
INFO: current trust_remote_code: False
INFO: current using single node, use input chip_num 4
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING]
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] *****************************************
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] *****************************************
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
[2025-02-25 08:56:43,578] [46740] [281472869855248] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,582] [46740] [281472869855248] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,582] [46740] [281472869855248] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,583] [46740] [281472869855248] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,661] [46739] [281472872112144] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,666] [46739] [281472872112144] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,666] [46739] [281472872112144] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,667] [46739] [281472872112144] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,667] [46737] [281473370341392] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,671] [46737] [281473370341392] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,671] [46737] [281473370341392] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,673] [46737] [281473370341392] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,733] [46738] [281473824780304] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,737] [46738] [281473824780304] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,737] [46738] [281473824780304] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,739] [46738] [281473824780304] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:47,496] [46737] [281473370341392] [llm] [INFO][logging.py-227] : Skip binding cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2025-02-25 08:56:48,369] [46737] [281473370341392] [llm] [INFO][logging.py-227] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[W compiler_depend.ts:733] Warning: expandable_segments setting failure, now change to `False`. (function operator())
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[2025-02-25 08:56:56,693] [46738] [281473824780304] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
[2025-02-25 08:56:56,704] [46737] [281473370341392] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
.................
[2025-02-25 08:57:40,720] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ---------------end inference---------------
2%|▏ | 59/3270 [00:13<11:52, 4.51it/s][2025-02-25 08:57:40.751400] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x12424105ec00, free fail]
[2025-02-25 08:57:40.751522] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x124241187c00, free fail
[2025-02-25 08:57:40.751547] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x124241195400, free fail
[2025-02-25 08:57:40.755517] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x12424105ec00, free fail
.......................
[2025-02-25 09:07:45,701] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ---------------begin inference---------------
[2025-02-25 09:07:45,740] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ------total req num: 1, infer start--------
[2025-02-25 09:07:46.193531] [error] [46740] [rt_backend_module.cpp:192] rt KernelLaunch With Flag fail, error:207001
[2025-02-25 09:07:46.194281] [error] [46740] [kernel_base.cpp:334] Mki RtFunction LaunchWithFlag fail
[2025-02-25 09:07:46.194301] [error] [46740] [mki_node_implement.cpp:212] LinearOpsRunner_19_0_0_0_1_0[1] PpMatMulNzF16Kernel run fail, launchParam
:specificParam: {"enDequant":false,"oriShape":[106,4096,1536],"tilingK":0,"tilingN":0,"transposeA":false,"transposeB":true,"withBias":false}
intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 256, 112, 16], dataSize:917504, data:1
intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 256, 1536, 16], dataSize:12582912, data:1
outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 96, 112, 16], dataSize:344064, data:1
st: code:7, msg:Mki RtFunction Launch fail
[2025-02-25 09:07:46.194440] [error] [46740] [ops_runner.cpp:602] RunKernel failed! ret:3
[2025-02-25 09:07:46.194451] [error] [46740] [ops_runner.cpp:397] RunAllKernel failed! ret:3
[2025-02-25 09:07:46.194462] [error] [46740] [runner.cpp:115] LinearOpsRunner_19_0_0_0_1_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194480] [error] [46740] [graph_runner.cpp:906] LinearNoQuantRunner_19_0_0_0_1:2708 node[0] execute fail, runner name:LinearOpsRunner
[2025-02-25 09:07:46.194497] [error] [46740] [runner.cpp:115] LinearNoQuantRunner_19_0_0_0_1:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194510] [error] [46740] [graph_runner.cpp:906] NormLinearRunner_19_0_0_0:2708 node[1] execute fail, runner name:LinearNoQuantRunner
[2025-02-25 09:07:46.194524] [error] [46740] [runner.cpp:115] NormLinearRunner_19_0_0_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194537] [error] [46740] [graph_runner.cpp:906] QKVLinearSplitPackRunner_19_0_0:2708 node[0] execute fail, runner name:NormLinearRunner
[2025-02-25 09:07:46.194552] [error] [46740] [runner.cpp:115] QKVLinearSplitPackRunner_19_0_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194565] [error] [46740] [graph_runner.cpp:906] AttentionRunner_19_0:2708 node[0] execute fail, runner name:QKVLinearSplitPackRunner
[2025-02-25 09:07:46.194579] [error] [46740] [runner.cpp:115] AttentionRunner_19_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194591] [error] [46740] [graph_runner.cpp:906] Prefill_layerRunner_19:2708 node[0] execute fail, runner name:AttentionRunner
[2025-02-25 09:07:46.194605] [error] [46740] [runner.cpp:115] Prefill_layerRunner_19:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194626] [error] [46740] [operation_base.cpp:726] Prefill_layer_19 execute Prefill_layerRunner fail
[2025-02-25 09:07:46.194637] [error] [46740] [operation_base.cpp:764] Prefill_layer_19 execute fail, error code: 3
[2025-02-25 09:07:46,195] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 286, in infer
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 430, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 338, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 270, in generate_token
logits = model.forward(
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 208, in forward
return self.model.forward(**kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 466, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
File "/usr/local/Ascend/atb-models/atb_llm/models/llama/flash_causal_llama.py", line 661, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
[2025-02-25 09:07:46,250] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 46740] 2025-02-25-09:07:46.130.376 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
The error from device(0), serial number is 6, there is an aicore error, core id is 0, error code = 0x800000, dump info: pc start: 0x1000124080277938, current: 0x124080278c44, vec error info: 0xfa3fffd, mte error info: 0x301c752, ifu error info: 0x2f41a22763680, ccu error info: 0, cube error info: 0x49, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x1240c05a8c00, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:639]
.......................................
DEVICE[3] PID[46740]:
EXCEPTION STREAM:
Exception info:TGID=47749, model id=65535, stream id=0, stream phase=3
Message info[0]:RTS_HWTS: aicore exception, slot_id=1, stream_id=0
Other info[0]:time=2025-02-25-09:07:46.981.238, function=int_process_hwts_task_exception, line=1704, error code=0x26
[2025-02-25 09:07:46,290] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.288.052 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.288.052 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
[2025-02-25 09:07:46,328] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.325.754 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.325.754 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
[2025-02-25 09:07:46,364] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.361.654 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
...........................
[ERROR] 2025-02-25-09:08:08 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:08:08.047.233 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
100%|██████████| 3270/3270 [10:40<00:00, 5.11it/s]
global: 100%|██████████| 1/1 [10:40<00:00, 640.55s/it]
[2025-02-25 09:08:08,056] [46740] [281472869855248] [llm] [INFO][model_test.py-900] : precision test end
[2025-02-25 09:11:02,187] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -9) local_rank: 0 (pid: 46737) of binary: /root/miniconda3/envs/Python310/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/Python310/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 1 (local_rank: 1)
exitcode : -9 (pid: 46738)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46738
[2]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 2 (local_rank: 2)
exitcode : -9 (pid: 46739)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46739
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 46737)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46737
============================================================
点赞
收藏
回复
分享
举报
已解决
nohup: ignoring input
INFO: Detected Ascend NPU
INFO: current model_type: pa_fp16
INFO: current test_mode: full_BoolQ
INFO: current batch_size: 1
INFO: current model_name: llama
INFO: current weight_dir: /root/model_pt/DeepSeek-R1-Distill-Llama-8B/
INFO: current model_type: pa
INFO: current data_type: fp16
INFO: current test_mode: full
INFO: current dataset: BoolQ
INFO: current trust_remote_code: False
INFO: current using single node, use input chip_num 4
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING]
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] *****************************************
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2025-02-25 08:56:36,486] torch.distributed.run: [WARNING] *****************************************
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/Python310/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
[2025-02-25 08:56:43,578] [46740] [281472869855248] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,582] [46740] [281472869855248] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,582] [46740] [281472869855248] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,583] [46740] [281472869855248] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,661] [46739] [281472872112144] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,666] [46739] [281472872112144] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,666] [46739] [281472872112144] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,667] [46739] [281472872112144] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,667] [46737] [281473370341392] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,671] [46737] [281473370341392] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,671] [46737] [281473370341392] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,673] [46737] [281473370341392] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:43,733] [46738] [281473824780304] [llm] [INFO][model_test.py-522] :
model_name: llama3.1_8b
model_type: pa
data_type: fp16
test_mode: full
data_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/data/NPU/precision_test/full/fp16/llama3.1_8b
time_limit: 0
batch_size: 1
result_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/results/NPU/precision_test/full/fp16/llama3.1_8b
log_dir: /usr/local/Ascend/atb-models/tests/modeltest/outputs/logs
[2025-02-25 08:56:43,737] [46738] [281473824780304] [llm] [INFO][model_test.py-541] : ATB env get success.
[2025-02-25 08:56:43,737] [46738] [281473824780304] [llm] [INFO][model_test.py-546] : ATB_SPEED env get success
[2025-02-25 08:56:43,739] [46738] [281473824780304] [llm] [INFO][model_test.py-859] : precision test start
[2025-02-25 08:56:47,496] [46737] [281473370341392] [llm] [INFO][logging.py-227] : Skip binding cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2025-02-25 08:56:48,369] [46737] [281473370341392] [llm] [INFO][logging.py-227] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[W compiler_depend.ts:733] Warning: expandable_segments setting failure, now change to `False`. (function operator())
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),compiler_depend.ts:732:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[2025-02-25 08:56:56,693] [46738] [281473824780304] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
[2025-02-25 08:56:56,704] [46737] [281473370341392] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
.................
[2025-02-25 08:57:40,720] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ---------------end inference---------------
2%|▏ | 59/3270 [00:13<11:52, 4.51it/s][2025-02-25 08:57:40.751400] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x12424105ec00, free fail]
[2025-02-25 08:57:40.751522] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x124241187c00, free fail
[2025-02-25 08:57:40.751547] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x124241195400, free fail
[2025-02-25 08:57:40.755517] [error] [46738] [block_mem_allocation_solver.cpp:63] can't find block: 0x12424105ec00, free fail
.......................
[2025-02-25 09:07:45,701] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ---------------begin inference---------------
[2025-02-25 09:07:45,740] [46737] [281473370341392] [llm] [INFO][logging.py-227] : ------total req num: 1, infer start--------
[2025-02-25 09:07:46.193531] [error] [46740] [rt_backend_module.cpp:192] rt KernelLaunch With Flag fail, error:207001
[2025-02-25 09:07:46.194281] [error] [46740] [kernel_base.cpp:334] Mki RtFunction LaunchWithFlag fail
[2025-02-25 09:07:46.194301] [error] [46740] [mki_node_implement.cpp:212] LinearOpsRunner_19_0_0_0_1_0[1] PpMatMulNzF16Kernel run fail, launchParam
:specificParam: {"enDequant":false,"oriShape":[106,4096,1536],"tilingK":0,"tilingN":0,"transposeA":false,"transposeB":true,"withBias":false}
intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 256, 112, 16], dataSize:917504, data:1
intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 256, 1536, 16], dataSize:12582912, data:1
outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 96, 112, 16], dataSize:344064, data:1
st: code:7, msg:Mki RtFunction Launch fail
[2025-02-25 09:07:46.194440] [error] [46740] [ops_runner.cpp:602] RunKernel failed! ret:3
[2025-02-25 09:07:46.194451] [error] [46740] [ops_runner.cpp:397] RunAllKernel failed! ret:3
[2025-02-25 09:07:46.194462] [error] [46740] [runner.cpp:115] LinearOpsRunner_19_0_0_0_1_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194480] [error] [46740] [graph_runner.cpp:906] LinearNoQuantRunner_19_0_0_0_1:2708 node[0] execute fail, runner name:LinearOpsRunner
[2025-02-25 09:07:46.194497] [error] [46740] [runner.cpp:115] LinearNoQuantRunner_19_0_0_0_1:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194510] [error] [46740] [graph_runner.cpp:906] NormLinearRunner_19_0_0_0:2708 node[1] execute fail, runner name:LinearNoQuantRunner
[2025-02-25 09:07:46.194524] [error] [46740] [runner.cpp:115] NormLinearRunner_19_0_0_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194537] [error] [46740] [graph_runner.cpp:906] QKVLinearSplitPackRunner_19_0_0:2708 node[0] execute fail, runner name:NormLinearRunner
[2025-02-25 09:07:46.194552] [error] [46740] [runner.cpp:115] QKVLinearSplitPackRunner_19_0_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194565] [error] [46740] [graph_runner.cpp:906] AttentionRunner_19_0:2708 node[0] execute fail, runner name:QKVLinearSplitPackRunner
[2025-02-25 09:07:46.194579] [error] [46740] [runner.cpp:115] AttentionRunner_19_0:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194591] [error] [46740] [graph_runner.cpp:906] Prefill_layerRunner_19:2708 node[0] execute fail, runner name:AttentionRunner
[2025-02-25 09:07:46.194605] [error] [46740] [runner.cpp:115] Prefill_layerRunner_19:2709 Execute Failed. st: 3
[2025-02-25 09:07:46.194626] [error] [46740] [operation_base.cpp:726] Prefill_layer_19 execute Prefill_layerRunner fail
[2025-02-25 09:07:46.194637] [error] [46740] [operation_base.cpp:764] Prefill_layer_19 execute fail, error code: 3
[2025-02-25 09:07:46,195] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 286, in infer
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 430, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 338, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 270, in generate_token
logits = model.forward(
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 208, in forward
return self.model.forward(**kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 466, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
File "/usr/local/Ascend/atb-models/atb_llm/models/llama/flash_causal_llama.py", line 661, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
[2025-02-25 09:07:46,250] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 46740] 2025-02-25-09:07:46.130.376 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
The error from device(0), serial number is 6, there is an aicore error, core id is 0, error code = 0x800000, dump info: pc start: 0x1000124080277938, current: 0x124080278c44, vec error info: 0xfa3fffd, mte error info: 0x301c752, ifu error info: 0x2f41a22763680, ccu error info: 0, cube error info: 0x49, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x1240c05a8c00, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:639]
.......................................
DEVICE[3] PID[46740]:
EXCEPTION STREAM:
Exception info:TGID=47749, model id=65535, stream id=0, stream phase=3
Message info[0]:RTS_HWTS: aicore exception, slot_id=1, stream_id=0
Other info[0]:time=2025-02-25-09:07:46.981.238, function=int_process_hwts_task_exception, line=1704, error code=0x26
[2025-02-25 09:07:46,290] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.288.052 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.288.052 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
[2025-02-25 09:07:46,328] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.325.754 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.325.754 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
[2025-02-25 09:07:46,364] [46740] [281472869855248] [llm] [ERROR][logging.py-43] : Error occurred npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:07:46.361.654 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2265, in __run_full_dataset_boolq
_, _, _ = self.pa_runner.infer(infer_input, local_batch_size, 1, False, self.is_chat_model)
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer
torch.npu.synchronize()
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 34, in synchronize
return torch_npu._C._npu_synchronize()
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:445 NPU function error: aclrtSynchronizeDevice, error code is 507015
[ERROR] 2025-02-25-09:07:46 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
...........................
[ERROR] 2025-02-25-09:08:08 (PID:46740, Device:3, RankID:3) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 46740] 2025-02-25-09:08:08.047.233 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
100%|██████████| 3270/3270 [10:40<00:00, 5.11it/s]
global: 100%|██████████| 1/1 [10:40<00:00, 640.55s/it]
[2025-02-25 09:08:08,056] [46740] [281472869855248] [llm] [INFO][model_test.py-900] : precision test end
[2025-02-25 09:11:02,187] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -9) local_rank: 0 (pid: 46737) of binary: /root/miniconda3/envs/Python310/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/Python310/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 1 (local_rank: 1)
exitcode : -9 (pid: 46738)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46738
[2]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 2 (local_rank: 2)
exitcode : -9 (pid: 46739)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46739
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-25_09:11:02
host : bms-41ba-0002
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 46737)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 46737
============================================================