【版本信息】AscendHDK 24.1.rc2，CANN 8.0.RC2，MindIE 1.0.RC2,银河麒麟GFB
【问题描述】使用MindIE推理千问1.5-0.5B大模型报错，报错信息如下，报错日志见附件，出现EZ9999: Inner Error!
【报错信息】
1970-01-01 12:03:44,967 [INFO] [pid: 1458] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': '0', 'use_host_chooser': True, 'bind_cpu': True}
1970-01-01 12:03:45,975 [INFO] [pid: 1458] logging.py-53: Skip binding cpu.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
1970-01-01 12:03:49,001 [INFO] [pid: 1458] logging.py-53: model_runner.quantize: None
, model_runner.kv_quant: None
, model_runner.dytpe: torch.float16
1970-01-01 12:03:49,002 [INFO] [pid: 1458] logging.py-53: Rank table file location:
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
1970-01-01 12:04:13,710 [INFO] [pid: 1458] dist.py-94: initialize_distributed has been Set
1970-01-01 12:04:13,712 [INFO] [pid: 1458] logging.py-53: init tokenizer done: Qwen2TokenizerFast(name_or_path='/root/llm/qwen1.5-0.5b', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
1970-01-01 12:04:15,622 [INFO] [pid: 1458] causal_lm.py-57: NPUSocInfo(soc_name='', soc_version=202, need_nz=True)
1970-01-01 12:04:15,674 [INFO] [pid: 1458] causal_qwen2.py-50: >>>> qwen1.5_flash_attention_model is called.
EZ9999: Inner Error!
EZ9999: 1970-01-01-12:04:28.725.671 The error from device(0), serial number is 4, there is an aicore error, core id is 6, error code = 0x800000, dump info: pc start: 0x8001240000000ec, current: 0x1240000001b0, vec error info: 0x7799cf7, mte error info: 0x3000069, ifu error info: 0x2572d2ac5c780, ccu error info: 0xdfaeb6bd0018dd1a, cube error info: 0x8c, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x12c0406ed000, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:532]
TraceBack (most recent call last):
The extend info from device(0), serial number is 4, there is aicore error, core id is 6, aicore int: 0x10, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101d14000000000, biu status1: 0x80000201020000, clk gate mask: 0, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0x3580000000000000, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0, dbg data3: 0, dfx data: 0xe7[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:563]
The dha(mata) info from device(0), dha id is 0, dha status 1 info:0x23[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:653]
The dha(mata) info from device(0), dha id is 1, dha status 1 info:0x3[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:653]
The device(0), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:586]
coreId( 0): 0x800000 [FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:600]
Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1220]
Aicore kernel execute failed, device_id=0, stream_id=13, report_stream_id=13, task_id=2, flip_num=0, fault kernel_name=ZerosLike_d2db1a80c523e7e59a032c95969880af_high_performance_210000000, fault kernel info ext=none, program id=0, hash=1637811349845522574.[FUNC:GetError][FILE:stream.cc][LINE:1082]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1082]
rtStreamSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[1458]:
EXCEPTION STREAM:
Exception info:TGID=30336, model id=65535, stream id=13, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=13
Other info[0]:time=1970-01-01-12:04:28.687.742, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/Ascend/llm_model/examples/run_fa.py", line 224, in <module>
fa_runner = FARunner(**input_dict)
File "/usr/local/Ascend/llm_model/examples/run_fa.py", line 92, in __init__
self.model.load_weights()
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 84, in load_weights
self.model = self.model_cls(self.config, weights)
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/causal_qwen2.py", line 37, in __init__
self.lm_head_indices_fake = torch.tensor([0], dtype=torch.int64, device=CAUSAL_QWEN2_DEVICE_NAME)
RuntimeError: ACL stream synchronize failed, error code:507015
[W compiler_depend.ts:409] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 1970-01-01-12:04:28.767.691 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W compiler_depend.ts:392] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 1970-01-01-12:04:28.797.258 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[ERROR] 1970-01-01-12:04:31 (PID:1458, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
[1970-01-01 12:04:40,286] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1458) of binary: /usr/local/python3.10.2/bin/python3.10
Traceback (most recent call last):
File "/usr/local//python3.10.2/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_fa FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 1970-01-01_12:04:40
host : hiden-pc
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1458)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
点赞
收藏
回复
分享
举报
已解决
【版本信息】AscendHDK 24.1.rc2，CANN 8.0.RC2，MindIE 1.0.RC2,银河麒麟GFB
【问题描述】使用MindIE推理千问1.5-0.5B大模型报错，报错信息如下，报错日志见附件，出现EZ9999: Inner Error!
【报错信息】
1970-01-01 12:03:44,967 [INFO] [pid: 1458] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': '0', 'use_host_chooser': True, 'bind_cpu': True}
1970-01-01 12:03:45,975 [INFO] [pid: 1458] logging.py-53: Skip binding cpu.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
1970-01-01 12:03:49,001 [INFO] [pid: 1458] logging.py-53: model_runner.quantize: None
, model_runner.kv_quant: None
, model_runner.dytpe: torch.float16
1970-01-01 12:03:49,002 [INFO] [pid: 1458] logging.py-53: Rank table file location:
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
1970-01-01 12:04:13,710 [INFO] [pid: 1458] dist.py-94: initialize_distributed has been Set
1970-01-01 12:04:13,712 [INFO] [pid: 1458] logging.py-53: init tokenizer done: Qwen2TokenizerFast(name_or_path='/root/llm/qwen1.5-0.5b', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
1970-01-01 12:04:15,622 [INFO] [pid: 1458] causal_lm.py-57: NPUSocInfo(soc_name='', soc_version=202, need_nz=True)
1970-01-01 12:04:15,674 [INFO] [pid: 1458] causal_qwen2.py-50: >>>> qwen1.5_flash_attention_model is called.
EZ9999: Inner Error!
EZ9999: 1970-01-01-12:04:28.725.671 The error from device(0), serial number is 4, there is an aicore error, core id is 6, error code = 0x800000, dump info: pc start: 0x8001240000000ec, current: 0x1240000001b0, vec error info: 0x7799cf7, mte error info: 0x3000069, ifu error info: 0x2572d2ac5c780, ccu error info: 0xdfaeb6bd0018dd1a, cube error info: 0x8c, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x12c0406ed000, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:532]
TraceBack (most recent call last):
The extend info from device(0), serial number is 4, there is aicore error, core id is 6, aicore int: 0x10, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101d14000000000, biu status1: 0x80000201020000, clk gate mask: 0, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0x3580000000000000, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0, dbg data3: 0, dfx data: 0xe7[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:563]
The dha(mata) info from device(0), dha id is 0, dha status 1 info:0x23[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:653]
The dha(mata) info from device(0), dha id is 1, dha status 1 info:0x3[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:653]
The device(0), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:586]
coreId( 0): 0x800000 [FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:600]
Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1220]
Aicore kernel execute failed, device_id=0, stream_id=13, report_stream_id=13, task_id=2, flip_num=0, fault kernel_name=ZerosLike_d2db1a80c523e7e59a032c95969880af_high_performance_210000000, fault kernel info ext=none, program id=0, hash=1637811349845522574.[FUNC:GetError][FILE:stream.cc][LINE:1082]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1082]
rtStreamSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[1458]:
EXCEPTION STREAM:
Exception info:TGID=30336, model id=65535, stream id=13, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=13
Other info[0]:time=1970-01-01-12:04:28.687.742, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/Ascend/llm_model/examples/run_fa.py", line 224, in <module>
fa_runner = FARunner(**input_dict)
File "/usr/local/Ascend/llm_model/examples/run_fa.py", line 92, in __init__
self.model.load_weights()
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 84, in load_weights
self.model = self.model_cls(self.config, weights)
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/causal_qwen2.py", line 37, in __init__
self.lm_head_indices_fake = torch.tensor([0], dtype=torch.int64, device=CAUSAL_QWEN2_DEVICE_NAME)
RuntimeError: ACL stream synchronize failed, error code:507015
[W compiler_depend.ts:409] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 1970-01-01-12:04:28.767.691 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W compiler_depend.ts:392] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 1970-01-01-12:04:28.797.258 wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[ERROR] 1970-01-01-12:04:31 (PID:1458, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
[1970-01-01 12:04:40,286] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1458) of binary: /usr/local/python3.10.2/bin/python3.10
Traceback (most recent call last):
File "/usr/local//python3.10.2/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_fa FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 1970-01-01_12:04:40
host : hiden-pc
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1458)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================