2024-12-21 15:15:28,558 [INFO] [pid: 28544] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2024-12-21 15:15:32,938 [INFO] [pid: 28542] logging.py-180: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.735 [kernel_mgr.cpp:227][NNOP][ParseDynamicKernelConfig][29254] errno[561108] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] Parse dynamic kernel config fail.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.940 [kernel_workspace.cpp:90][NNOP][GetWorkspace][29254] errno[561000] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] AclOpKernelInit failed opType
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.994 [zero_op.cpp:62][NNOP][ZerosLikeAiCore][29254] errno[561118] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.084.022 [aclnn_zero.cpp:93][NNOP][aclnnInplaceZeroGetWorkspaceSize][29254] errno[561000] OpName:[aclnnInplaceZero_3] check zeroOut != nullptr failed.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.084.035 [aclnn_zero.cpp:164][NNOP][~UniqueExecutor][29254] errno[561102] OpName:[aclnnInplaceZero_3] When aclnnInplaceZeroGetWorkspaceSize do success, ReleaseTo(executor) should be called before return.
2024-12-21 15:15:33,089 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28542] 2024-12-21-15:15:33.083.805 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28542, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28542] 2024-12-21-15:15:33.083.805 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28542, Device:0, RankID:-1) ERR01100 OPS call acl api failed
2024-12-21 15:15:33,093 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.025 [kernel_mgr.cpp:227][NNOP][ParseDynamicKernelConfig][29256] errno[561108] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] Parse dynamic kernel config fail.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.177 [kernel_workspace.cpp:90][NNOP][GetWorkspace][29256] errno[561000] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] AclOpKernelInit failed opType
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.225 [zero_op.cpp:62][NNOP][ZerosLikeAiCore][29256] errno[561118] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.254 [aclnn_zero.cpp:93][NNOP][aclnnInplaceZeroGetWorkspaceSize][29256] errno[561000] OpName:[aclnnInplaceZero_3] check zeroOut != nullptr failed.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.266 [aclnn_zero.cpp:164][NNOP][~UniqueExecutor][29256] errno[561102] OpName:[aclnnInplaceZero_3] When aclnnInplaceZeroGetWorkspaceSize do success, ReleaseTo(executor) should be called before return.
2024-12-21 15:15:33,886 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28544] 2024-12-21-15:15:33.882.083 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28544, Device:1, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28544] 2024-12-21-15:15:33.882.083 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28544, Device:1, RankID:-1) ERR01100 OPS call acl api failed
2024-12-21 15:15:33,889 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2024-12-21 15:15:33.940539 28270 error [llm_infer_model_instance.cpp:45] Initialize modelBackends_ failed.
2024-12-21 15:15:33.940666 28270 error [llm_infer_model.cpp:20] Init model instance failed.
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
已杀死 
点赞
收藏
回复
分享
举报
已解决
2024-12-21 15:15:28,558 [INFO] [pid: 28544] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2024-12-21 15:15:32,938 [INFO] [pid: 28542] logging.py-180: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.735 [kernel_mgr.cpp:227][NNOP][ParseDynamicKernelConfig][29254] errno[561108] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] Parse dynamic kernel config fail.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.940 [kernel_workspace.cpp:90][NNOP][GetWorkspace][29254] errno[561000] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] AclOpKernelInit failed opType
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.083.994 [zero_op.cpp:62][NNOP][ZerosLikeAiCore][29254] errno[561118] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.084.022 [aclnn_zero.cpp:93][NNOP][aclnnInplaceZeroGetWorkspaceSize][29254] errno[561000] OpName:[aclnnInplaceZero_3] check zeroOut != nullptr failed.
[ERROR] OP(28542,mindie_llm_backend_connector):2024-12-21-15:15:33.084.035 [aclnn_zero.cpp:164][NNOP][~UniqueExecutor][29254] errno[561102] OpName:[aclnnInplaceZero_3] When aclnnInplaceZeroGetWorkspaceSize do success, ReleaseTo(executor) should be called before return.
2024-12-21 15:15:33,089 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28542] 2024-12-21-15:15:33.083.805 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28542, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28542] 2024-12-21-15:15:33.083.805 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28542, Device:0, RankID:-1) ERR01100 OPS call acl api failed
2024-12-21 15:15:33,093 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.025 [kernel_mgr.cpp:227][NNOP][ParseDynamicKernelConfig][29256] errno[561108] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] Parse dynamic kernel config fail.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.177 [kernel_workspace.cpp:90][NNOP][GetWorkspace][29256] errno[561000] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] AclOpKernelInit failed opType
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.225 [zero_op.cpp:62][NNOP][ZerosLikeAiCore][29256] errno[561118] OpName:[aclnnInplaceZero_3_ZerosLikeAiCore] ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.254 [aclnn_zero.cpp:93][NNOP][aclnnInplaceZeroGetWorkspaceSize][29256] errno[561000] OpName:[aclnnInplaceZero_3] check zeroOut != nullptr failed.
[ERROR] OP(28544,mindie_llm_backend_connector):2024-12-21-15:15:33.882.266 [aclnn_zero.cpp:164][NNOP][~UniqueExecutor][29256] errno[561102] OpName:[aclnnInplaceZero_3] When aclnnInplaceZeroGetWorkspaceSize do success, ReleaseTo(executor) should be called before return.
2024-12-21 15:15:33,886 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28544] 2024-12-21-15:15:33.882.083 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28544, Device:1, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/mindie/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/mindie/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 28544] 2024-12-21-15:15:33.882.083 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2024-12-21-15:15:33 (PID:28544, Device:1, RankID:-1) ERR01100 OPS call acl api failed
2024-12-21 15:15:33,889 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2024-12-21 15:15:33.940539 28270 error [llm_infer_model_instance.cpp:45] Initialize modelBackends_ failed.
2024-12-21 15:15:33.940666 28270 error [llm_infer_model.cpp:20] Init model instance failed.
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
已杀死