执行命令：
(Python310) root@bms-41ba-0002:/usr/local/Ascend/atb-models/tests/modeltest# bash run.sh pa_fp16 full_BoolQ 1 llama /root/model_pt/DeepSeek-R1-Distill-Llama-8B/ 8
报错：
...........
[2025-02-18 15:13:28,415] [2012] [281473381810192] [llm] [INFO][logging.py-227] : trans to 29
[2025-02-18 15:13:29,058] [2016] [281473110380560] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,134] [2012] [281473381810192] [llm] [INFO][logging.py-227] : <<<<<<< ori k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,142] [2012] [281473381810192] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,143] [2012] [281473381810192] [llm] [INFO][logging.py-227] : >>>>>>id of kcache is 281470481870864 id of vcache is 281470481874784
[2025-02-18 15:13:29,832] [2018] [281473778475024] [llm] [INFO][model_test.py-3093] : 6 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:6, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=6, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:29,833] [2018] [281473778475024] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:30,423] [2013] [281473158561808] [llm] [INFO][model_test.py-3093] : 1 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:1, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=1, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:30,425] [2013] [281473158561808] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:30,512] [2018] [281473778475024] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:31,089] [2019] [281473768980496] [llm] [INFO][model_test.py-3093] : 7 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:7, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=7, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15769534464,
[2025-02-18 15:13:31,091] [2019] [281473768980496] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:31,115] [2013] [281473158561808] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:31,760] [2019] [281473768980496] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:33,041] [2015] [281473563287568] [llm] [INFO][model_test.py-3093] : 3 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:3, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=3, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15769534464,
[2025-02-18 15:13:33,042] [2015] [281473563287568] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:33,713] [2014] [281473206050832] [llm] [INFO][model_test.py-3093] : 2 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:2, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=2, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:33,714] [2014] [281473206050832] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:33,732] [2015] [281473563287568] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:34,417] [2014] [281473206050832] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
[2025-02-18 15:13:38,324] [2012] [281473381810192] [llm] [INFO][logging.py-227] : warmup_memory(GB): 1.33
[2025-02-18 15:13:38,325] [2012] [281473381810192] [llm] [INFO][logging.py-227] : ---------------end warm_up---------------
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,339] [2014] [281473206050832] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py", line 85, in <module>
main()
File "/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py", line 82, in main
LlamaModelTest.create_instance()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 337, in create_instance
test_instance.run()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 346, in run
self.__run_multibs()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 398, in __run_multibs
self.__run_single_bs()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 432, in __run_single_bs
self.__run()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 571, in __run
self.__run_precision()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 893, in __run_precision
dataset_eval_func()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2308, in __run_full_dataset_boolq
total = ["total", correct_total / sum_total, correct_total, sum_total]
ZeroDivisionError: division by zero
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,349] [2013] [281473158561808] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,359] [2015] [281473563287568] [llm] [INFO][model_test.py-900] : precision test end
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,432] [2016] [281473110380560] [llm] [INFO][model_test.py-900] : precision test end
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,446] [2018] [281473778475024] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,451] [2017] [281473840082960] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,455] [2019] [281473768980496] [llm] [INFO][model_test.py-900] : precision test end
[ERROR] 2025-02-18-15:13:45 (PID:2012, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
[2025-02-18 15:13:55,023] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2017 closing signal SIGTERM
[2025-02-18 15:13:56,789] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 2012) of binary: /root/miniconda3/envs/Python310/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/Python310/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-18_15:13:55
host : bms-41ba-0002
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2012)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
点赞
收藏
回复
分享
举报
已解决
执行命令：
(Python310) root@bms-41ba-0002:/usr/local/Ascend/atb-models/tests/modeltest# bash run.sh pa_fp16 full_BoolQ 1 llama /root/model_pt/DeepSeek-R1-Distill-Llama-8B/ 8
报错：
...........
[2025-02-18 15:13:28,415] [2012] [281473381810192] [llm] [INFO][logging.py-227] : trans to 29
[2025-02-18 15:13:29,058] [2016] [281473110380560] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,134] [2012] [281473381810192] [llm] [INFO][logging.py-227] : <<<<<<< ori k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,142] [2012] [281473381810192] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:29,143] [2012] [281473381810192] [llm] [INFO][logging.py-227] : >>>>>>id of kcache is 281470481870864 id of vcache is 281470481874784
[2025-02-18 15:13:29,832] [2018] [281473778475024] [llm] [INFO][model_test.py-3093] : 6 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:6, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=6, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:29,833] [2018] [281473778475024] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:30,423] [2013] [281473158561808] [llm] [INFO][model_test.py-3093] : 1 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:1, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=1, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:30,425] [2013] [281473158561808] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:30,512] [2018] [281473778475024] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:31,089] [2019] [281473768980496] [llm] [INFO][model_test.py-3093] : 7 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:7, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=7, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15769534464,
[2025-02-18 15:13:31,091] [2019] [281473768980496] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:31,115] [2013] [281473158561808] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:31,760] [2019] [281473768980496] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:33,041] [2015] [281473563287568] [llm] [INFO][model_test.py-3093] : 3 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:3, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=3, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15769534464,
[2025-02-18 15:13:33,042] [2015] [281473563287568] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:33,713] [2014] [281473206050832] [llm] [INFO][model_test.py-3093] : 2 pa_runner: PARunner(model_path=/root/model_pt/DeepSeek-R1-Distill-Llama-8B/, input_text=None, max_position_embeddings=3584, max_input_length=3072, max_output_length=512, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=1, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=4, num_kv_heads=1, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=32, device=npu:2, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=100, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=8, rank=2, pp_rank=0, pp_groups=[[0], [1], [2], [3], [4], [5], [6], [7]], micro_batch_size=1) ), cla_share_factor=1, , max_memory=15872294912,
[2025-02-18 15:13:33,714] [2014] [281473206050832] [llm] [INFO][cache.py-98] : kv cache will allocate 0.0546875GB memory
[2025-02-18 15:13:33,732] [2015] [281473563287568] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
[2025-02-18 15:13:34,417] [2014] [281473206050832] [llm] [INFO][flash_causal_llama.py-701] : <<<<<<<after transdata k_caches[0].shape=torch.Size([28, 8, 128, 16])
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
[2025-02-18 15:13:38,324] [2012] [281473381810192] [llm] [INFO][logging.py-227] : warmup_memory(GB): 1.33
[2025-02-18 15:13:38,325] [2012] [281473381810192] [llm] [INFO][logging.py-227] : ---------------end warm_up---------------
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,339] [2014] [281473206050832] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
Traceback (most recent call last):
File "/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py", line 85, in <module>
main()
File "/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py", line 82, in main
LlamaModelTest.create_instance()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 337, in create_instance
test_instance.run()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 346, in run
self.__run_multibs()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 398, in __run_multibs
self.__run_single_bs()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 432, in __run_single_bs
self.__run()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 571, in __run
self.__run_precision()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 893, in __run_precision
dataset_eval_func()
File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 2308, in __run_full_dataset_boolq
total = ["total", correct_total / sum_total, correct_total, sum_total]
ZeroDivisionError: division by zero
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,349] [2013] [281473158561808] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,359] [2015] [281473563287568] [llm] [INFO][model_test.py-900] : precision test end
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,432] [2016] [281473110380560] [llm] [INFO][model_test.py-900] : precision test end
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,446] [2018] [281473778475024] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,451] [2017] [281473840082960] [llm] [INFO][model_test.py-900] : precision test end
global: 0it [00:00, ?it/s]
[2025-02-18 15:13:38,455] [2019] [281473768980496] [llm] [INFO][model_test.py-900] : precision test end
[ERROR] 2025-02-18-15:13:45 (PID:2012, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
[2025-02-18 15:13:55,023] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2017 closing signal SIGTERM
[2025-02-18 15:13:56,789] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 2012) of binary: /root/miniconda3/envs/Python310/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/Python310/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/Python310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/usr/local/Ascend/atb-models/tests/modeltest/core/llama_test.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-18_15:13:55
host : bms-41ba-0002
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2012)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================