昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
使用MindIE进行模型推理时报错 acl_hidden_state = acl_model_out[0] IndexError: list index out of range
收藏回复举报
使用MindIE进行模型推理时报错 acl_hidden_state = acl_model_out[0] IndexError: list index out of range
t('forum.solved') 已解决
新人帖
发表于2024-06-05 15:14:44
0 查看

问题描述

使用 Atlas 800 9000 服务器，部署MindIE进行大模型推理时出现以下报错

acl_hidden_state = acl_model_out[0] IndexError: list index out of range

完整报错日志如下：

(mindie-server) root@huawei:/usr/local/Ascend/mindie/latest/mindie-service/bin# ./mindieservice_daemon 
Use user defined log path: /usr/local/Ascend/mindie/1.0.RC1/mindie-service/logs/mindservice.log
2024-06-05 06:04:50,658 [INFO] [pid: 1447703] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,732 [INFO] [pid: 1447700] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,767 [INFO] [pid: 1447699] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,790 [INFO] [pid: 1447698] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,861 [INFO] [pid: 1447704] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,886 [INFO] [pid: 1447701] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,892 [INFO] [pid: 1447702] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:50,911 [INFO] [pid: 1447705] env.py-46: {'use_ascend': True, 'max_memory_gb': None, 'reserved_memory_gb': 3, 'skip_warmup': False, 'visible_devices': None, 'use_host_chooser': True, 'bind_cpu': True}
2024-06-05 06:04:57,141 [INFO] [pid: 1447703] cpu_binding.py-206: rank_id: 5, device_id: 5, numa_id: 4, shard_devices: [1, 5], cpus: [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119]
2024-06-05 06:04:57,143 [INFO] [pid: 1447703] cpu_binding.py-231: process 1447703, new_affinity is [108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119], cpu count 12
2024-06-05 06:04:57,293 [INFO] [pid: 1447700] cpu_binding.py-206: rank_id: 2, device_id: 2, numa_id: 2, shard_devices: [2, 6], cpus: [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]
2024-06-05 06:04:57,296 [INFO] [pid: 1447700] cpu_binding.py-231: process 1447700, new_affinity is [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59], cpu count 12
2024-06-05 06:04:57,352 [INFO] [pid: 1447699] cpu_binding.py-206: rank_id: 1, device_id: 1, numa_id: 4, shard_devices: [1, 5], cpus: [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119]
2024-06-05 06:04:57,354 [INFO] [pid: 1447699] cpu_binding.py-231: process 1447699, new_affinity is [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107], cpu count 12
2024-06-05 06:04:57,512 [INFO] [pid: 1447698] cpu_binding.py-206: rank_id: 0, device_id: 0, numa_id: 6, shard_devices: [0, 4], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167]
2024-06-05 06:04:57,515 [INFO] [pid: 1447698] cpu_binding.py-231: process 1447698, new_affinity is [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155], cpu count 12
2024-06-05 06:04:57,615 [INFO] [pid: 1447701] cpu_binding.py-206: rank_id: 3, device_id: 3, numa_id: 0, shard_devices: [3, 7], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
2024-06-05 06:04:57,618 [INFO] [pid: 1447701] cpu_binding.py-231: process 1447701, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], cpu count 12
2024-06-05 06:04:57,642 [INFO] [pid: 1447704] cpu_binding.py-206: rank_id: 6, device_id: 6, numa_id: 2, shard_devices: [2, 6], cpus: [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]
2024-06-05 06:04:57,645 [INFO] [pid: 1447704] cpu_binding.py-231: process 1447704, new_affinity is [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], cpu count 12
2024-06-05 06:04:57,657 [INFO] [pid: 1447705] cpu_binding.py-206: rank_id: 7, device_id: 7, numa_id: 0, shard_devices: [3, 7], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
2024-06-05 06:04:57,658 [INFO] [pid: 1447702] cpu_binding.py-206: rank_id: 4, device_id: 4, numa_id: 6, shard_devices: [0, 4], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167]
2024-06-05 06:04:57,659 [INFO] [pid: 1447705] cpu_binding.py-231: process 1447705, new_affinity is [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], cpu count 12
2024-06-05 06:04:57,660 [INFO] [pid: 1447702] cpu_binding.py-231: process 1447702, new_affinity is [156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167], cpu count 12
2024-06-05 06:04:58,040 [INFO] [pid: 1447698] logging.py-55: model_runner.quantize: None
, model_runner.dytpe: torch.float16
2024-06-05 06:04:58,041 [INFO] [pid: 1447698] logging.py-55: Rank table file location: 
2024-06-05 06:05:25,665 [INFO] [pid: 1447703] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:26,251 [INFO] [pid: 1447700] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:26,450 [INFO] [pid: 1447699] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:26,961 [INFO] [pid: 1447698] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:26,966 [INFO] [pid: 1447698] logging.py-55: init tokenizer done: BaichuanTokenizer(name_or_path='/models/Baichuan2-13B-Chat', vocab_size=125696, model_max_length=8192, is_fast=False, padding_side='left', truncation_side='left', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
        0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=True, normalized=True, special=True),
        1: AddedToken("<s>", rstrip=False, lstrip=False, single_word=True, normalized=True, special=True),
        2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=True, normalized=True, special=True),
}
2024-06-05 06:05:27,037 [INFO] [pid: 1447704] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:27,098 [INFO] [pid: 1447705] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:27,111 [INFO] [pid: 1447698] logging.py-55: NPUSocInfo(soc_name='', soc_version=103, need_nz=False)
2024-06-05 06:05:27,765 [INFO] [pid: 1447701] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:27,800 [INFO] [pid: 1447702] dist.py-80: initialize_distributed has been Set
2024-06-05 06:05:39,069 [INFO] [pid: 1447698] logging.py-55: model:
 FlashBaichuanForCausalLM(
  (attn_mask): AttentionMask()
  (model): FlashBaichuanModel(
    (embed_tokens): TensorParallelEmbedding()
    (layers): ModuleList(
      (0-39): 40 x FlashBaichuanLayer(
        (self_attn): FlashBaichuanAttention(
          (rotary_emb): PositionRotaryEmbedding()
          (W_pack): TensorParallelColumnLinear(
            (linear): FastLinear()
          )
          (o_proj): TensorParallelRowLinear(
            (linear): FastLinear()
          )
        )
        (mlp): BaichuanMLP(
          (act): SiLU()
          (gate_up_proj): TensorParallelColumnLinear(
            (linear): FastLinear()
          )
          (down_proj): TensorParallelRowLinear(
            (linear): FastLinear()
          )
        )
        (input_layernorm): BaichuanRMSNorm()
        (post_attention_layernorm): BaichuanRMSNorm()
      )
    )
    (norm): BaichuanRMSNorm()
  )
  (lm_head): TensorParallelHead(
    (linear): FastLinear()
  )
  (ascend_atten_mask): AttentionMask()
)
2024-06-05 06:05:47,254 [INFO] [pid: 1447704] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:05:48,667 [INFO] [pid: 1447704] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:05:48,686 [WARNING] [pid: 1447704] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281473384347136 id of vcache is 281473384347216
2024-06-05 06:05:48,833 [INFO] [pid: 1447703] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:05:49,029 [INFO] [pid: 1447703] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:05:49,032 [WARNING] [pid: 1447703] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281472911245536 id of vcache is 281472911245616
2024-06-05 06:05:49,285 [INFO] [pid: 1447698] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:05:50,365 [INFO] [pid: 1447698] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:05:50,368 [WARNING] [pid: 1447698] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281473308126768 id of vcache is 281473308126848
2024-06-05 06:05:53,899 [INFO] [pid: 1447701] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:05:54,746 [INFO] [pid: 1447701] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:05:54,750 [WARNING] [pid: 1447701] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281472726843312 id of vcache is 281472728481856
2024-06-05 06:05:59,365 [INFO] [pid: 1447700] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:06:00,526 [INFO] [pid: 1447700] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:06:00,550 [WARNING] [pid: 1447700] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281472491184560 id of vcache is 281472491184640
2024-06-05 06:06:06,454 [INFO] [pid: 1447702] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:06:07,056 [INFO] [pid: 1447702] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:06:07,088 [WARNING] [pid: 1447702] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281472795629008 id of vcache is 281472795629088
2024-06-05 06:06:11,950 [INFO] [pid: 1447699] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:06:12,365 [INFO] [pid: 1447699] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:06:12,369 [WARNING] [pid: 1447699] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281473020617856 id of vcache is 281473020617936
2024-06-05 06:06:17,696 [INFO] [pid: 1447705] flash_causal_baichuan.py-141: >>>> quant-None
2024-06-05 06:06:17,744 [INFO] [pid: 1447705] flash_causal_baichuan.py-165: >>>> baichuan2_13b_PagedAttentionParam is inited.
2024-06-05 06:06:17,747 [WARNING] [pid: 1447705] flash_causal_baichuan.py-183: >>>>>>id of kcache is 281473184983424 id of vcache is 281473184983504
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[Model] >>> Exception:list index out of range
Traceback (most recent call last):
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/model.py", line 28, in initialize
    return self.python_model.initialize(config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/standard_model.py", line 24, in initialize
    self.model = WrapperFactory.get_model_wrapper(model_config)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/wrapper_factory.py", line 23, in get_model_wrapper
    return StandardATBModelWrapper(
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 71, in __init__
    self.warmup(max_prefill_tokens, max_seq_len, max_iter_times)
  File "/usr/local/Ascend/mindie/1.0.RC1/mindie-service/bin/utils/atb_wrapper.py", line 111, in warmup
    self.model_runner.forward(
  File "/work/packages/atb-models/atb_llm/runner/model_runner.py", line 81, in forward
    return self.model.forward(**kwargs)
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 427, in forward
    hidden_states = self.execute_ascend_operator(
  File "/work/packages/atb-models/atb_llm/models/baichuan/v2_13b/flash_causal_baichuan.py", line 341, in execute_ascend_operator
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range
[Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
free(): invalid pointer
Daemon is killing...
Killed

环境信息

硬件：

  • Atlas A800(9000)
  • 910ProB * 8

软件：

  • Ascend-hdk-910-npu-driver_23.0.0_linux-aarch64.run*
  • Ascend-hdk-910-npu-firmware_7.1.0.3.220.run*
  • Ascend-cann-kernels-910_8.0.RC1_linux.run*
  • Ascend-cann-toolkit_8.0.RC1_linux-aarch64.run*
  • python==3.10
  • apex-0.1_ascend_20240413-cp310-cp310-linux_aarch64.whl
  • torch==2.1.0
  • torch_npu==2.1.0.post3
  • Ascend-mindie_1.0.RC1_linux-aarch64.run*
  • Ascend-mindie-atb-models_1.0.RC1_linux-aarch64_torch2.1.0-abi0.tar.gz

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情