镜像版本：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.RC3-300I-Duo-arm64

执行命令：/usr/local/Ascend/mindie/latest/mindie-service/bin# ./mindieservice_daemon

报错截图 

 

2024-12-20 07:24:07,431 [INFO] [pid: 16711] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:07,745 [INFO] [pid: 16710] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:07,776 [INFO] [pid: 16709] logging.py-180: <<<<<<< ori k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:07,784 [INFO] [pid: 16709] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:07,785 [INFO] [pid: 16709] logging.py-180: >>>>>>id of kcache is 281471280697456 id of vcache is 281471280697360 

2024-12-20 07:24:08,773 [INFO] [pid: 16721] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:09,282 [INFO] [pid: 16713] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:10,516 [INFO] [pid: 16714] flash_causal_lm.py-281: <<<<<<<after transdata k_caches[0].shape=torch.Size([800, 32, 128, 16]) 

2024-12-20 07:24:12,151 [ERROR] model.py:39 - [Model]   >>> Exception:Setup fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.  

Traceback (most recent call last): 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize 

    return self.python_model.initialize(config) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize 

    self.generator = Generator( 

                     ^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__ 

    self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up 

    self.generator_backend.warm_up(model_inputs) 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up 

    _ = self.forward(model_inputs, **kwargs) 

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper 

    return func(*args, **kwargs) 

           ^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward 

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward 

    logits = self.forward_tensor( 

             ^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor 

    logits = self.model_runner.forward( 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward 

    return self.model.forward(**kwargs) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/llm_model/atb_llm/models/qwen/vl/flash_causal_qwen.py", line 283, in forward 

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 356, in execute_ascend_operator 

    acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param) 

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

RuntimeError: Setup fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.  

 

2024-12-20 07:24:12,154 [ERROR] model.py:42 - [Model]   >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'} 

2024-12-20 07:24:12,293 [ERROR] model.py:39 - [Model]   >>> Exception:Setup fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.