(base) root@84b3d83dca4d:/usr/local/Ascend/mindie/latest/mindie-service/logs# ../bin/mindieservice_daemon
Daemon wait pid with 24077, status 02024-08-05 00:13:18,327 [INFO] [pid: 24096] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-08-05 00:13:18,460 [INFO] [pid: 24097] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-08-05 00:13:20,126 [INFO] [pid: 24096] logging.py-53: Skip binding cpu.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-08-05 00:13:20,675 [INFO] [pid: 24096] logging.py-53: model_runner.quantize: None
, model_runner.kv_quant: None
, model_runner.dytpe: torch.float16
2024-08-05 00:13:20,675 [INFO] [pid: 24096] logging.py-53: Rank table file location:
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
2024-08-05 00:13:25,915 [INFO] [pid: 24096] dist.py-94: initialize_distributed has been Set
2024-08-05 00:13:25,917 [INFO] [pid: 24096] logging.py-53: init tokenizer done: Qwen2TokenizerFast(name_or_path='/root/qwen1.5-7B-Chat', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
2024-08-05 00:13:26,032 [INFO] [pid: 24096] logging.py-53: NPUSocInfo(soc_name='', soc_version=202, need_nz=True)
2024-08-05 00:13:26,059 [INFO] [pid: 24096] flash_causal_qwen2.py-52: >>>> qwen_DecoderModel is called.
2024-08-05 00:13:26,501 [INFO] [pid: 24097] dist.py-94: initialize_distributed has been Set
2024-08-05 00:13:26,706 [INFO] [pid: 24097] flash_causal_qwen2.py-52: >>>> qwen_DecoderModel is called.
2024-08-05 00:13:33,141 [INFO] [pid: 24096] logging.py-53: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-31): 32 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000fffbf001d9f0)
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000fffe9c01d9f0)
Daemon wait pid with 24096, status 134Daemon is killing...
Killed
(base) root@84b3d83dca4d:/usr/local/Ascend/mindie/latest/mindie-service/logs# tail -800f pythonlog.log.24096
2024-08-05 00:13:33,303 [ERROR] model.py:30 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: 2024-08-05-00:13:33.297.526 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
Op ZerosLike does not has any binary.
Kernel Run failed. opType: 3, ZerosLike
launch failed for ZerosLike, errno:561000.
[ERROR] 2024-08-05-00:13:33 (PID:24096, Device:0, RankID:-1) ERR01005 OPS internal error
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.10/site-packages/model_wrapper/model.py", line 28, in initialize
return self.python_model.initialize(config)
File "/root/anaconda3/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 28, in initialize
self.generator = Generator(
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 56, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_iter_times)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 133, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 42, in warm_up
_ = self.forward(model_inputs, **kwargs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 52, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 60, in forward
logits = self.forward_tensor(
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 86, in forward_tensor
logits = self.model_runner.forward(
File "/root/atb_llm/runner/model_runner.py", line 100, in forward
return self.model.forward(**kwargs)
File "/root/atb_llm/models/base/flash_causal_lm.py", line 223, in forward
self.init_ascend_weight()
File "/root/atb_llm/models/qwen2/flash_causal_qwen2.py", line 83, in init_ascend_weight
weight_wrapper = self.get_weights()
File "/root/atb_llm/models/qwen2/flash_causal_qwen2.py", line 69, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
File "/root/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: 2024-08-05-00:13:33.297.526 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
Op ZerosLike does not has any binary.
Kernel Run failed. opType: 3, ZerosLike
launch failed for ZerosLike, errno:561000.
[ERROR] 2024-08-05-00:13:33 (PID:24096, Device:0, RankID:-1) ERR01005 OPS internal error
2024-08-05 00:13:33,305 [ERROR] model.py:33 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
点赞
收藏
回复
分享
举报
已解决
(base) root@84b3d83dca4d:/usr/local/Ascend/mindie/latest/mindie-service/logs# ../bin/mindieservice_daemon
Daemon wait pid with 24077, status 02024-08-05 00:13:18,327 [INFO] [pid: 24096] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-08-05 00:13:18,460 [INFO] [pid: 24097] env.py-55: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-08-05 00:13:20,126 [INFO] [pid: 24096] logging.py-53: Skip binding cpu.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-08-05 00:13:20,675 [INFO] [pid: 24096] logging.py-53: model_runner.quantize: None
, model_runner.kv_quant: None
, model_runner.dytpe: torch.float16
2024-08-05 00:13:20,675 [INFO] [pid: 24096] logging.py-53: Rank table file location:
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
2024-08-05 00:13:25,915 [INFO] [pid: 24096] dist.py-94: initialize_distributed has been Set
2024-08-05 00:13:25,917 [INFO] [pid: 24096] logging.py-53: init tokenizer done: Qwen2TokenizerFast(name_or_path='/root/qwen1.5-7B-Chat', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
2024-08-05 00:13:26,032 [INFO] [pid: 24096] logging.py-53: NPUSocInfo(soc_name='', soc_version=202, need_nz=True)
2024-08-05 00:13:26,059 [INFO] [pid: 24096] flash_causal_qwen2.py-52: >>>> qwen_DecoderModel is called.
2024-08-05 00:13:26,501 [INFO] [pid: 24097] dist.py-94: initialize_distributed has been Set
2024-08-05 00:13:26,706 [INFO] [pid: 24097] flash_causal_qwen2.py-52: >>>> qwen_DecoderModel is called.
2024-08-05 00:13:33,141 [INFO] [pid: 24096] logging.py-53: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-31): 32 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000fffbf001d9f0)
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000fffe9c01d9f0)
Daemon wait pid with 24096, status 134Daemon is killing...
Killed
(base) root@84b3d83dca4d:/usr/local/Ascend/mindie/latest/mindie-service/logs# tail -800f pythonlog.log.24096
2024-08-05 00:13:33,303 [ERROR] model.py:30 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: 2024-08-05-00:13:33.297.526 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
Op ZerosLike does not has any binary.
Kernel Run failed. opType: 3, ZerosLike
launch failed for ZerosLike, errno:561000.
[ERROR] 2024-08-05-00:13:33 (PID:24096, Device:0, RankID:-1) ERR01005 OPS internal error
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.10/site-packages/model_wrapper/model.py", line 28, in initialize
return self.python_model.initialize(config)
File "/root/anaconda3/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 28, in initialize
self.generator = Generator(
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 56, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_iter_times)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 133, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 42, in warm_up
_ = self.forward(model_inputs, **kwargs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 52, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 60, in forward
logits = self.forward_tensor(
File "/root/anaconda3/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 86, in forward_tensor
logits = self.model_runner.forward(
File "/root/atb_llm/runner/model_runner.py", line 100, in forward
return self.model.forward(**kwargs)
File "/root/atb_llm/models/base/flash_causal_lm.py", line 223, in forward
self.init_ascend_weight()
File "/root/atb_llm/models/qwen2/flash_causal_qwen2.py", line 83, in init_ascend_weight
weight_wrapper = self.get_weights()
File "/root/atb_llm/models/qwen2/flash_causal_qwen2.py", line 69, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
File "/root/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: 2024-08-05-00:13:33.297.526 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
Op ZerosLike does not has any binary.
Kernel Run failed. opType: 3, ZerosLike
launch failed for ZerosLike, errno:561000.
[ERROR] 2024-08-05-00:13:33 (PID:24096, Device:0, RankID:-1) ERR01005 OPS internal error
2024-08-05 00:13:33,305 [ERROR] model.py:33 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}