部署qwen2.5vl-32b时,报错 Exception:call aclnnArange failed, detail:EZ9999: Inner Error!
收藏回复举报
部署qwen2.5vl-32b时,报错 Exception:call aclnnArange failed, detail:EZ9999: Inner Error!
t('forum.solved') 已解决
新人帖
发表于2025-07-24 10:42:04
0 查看

使用910b卡部署qwen2.5vl-32b时,无法拉起mindie服务

环境是true

部署过程为

https://modelers.cn/models/Models_Ecosystem/Qwen2.5-VL-32B-Instruct

拉起服务化时报

[2025-07-24 10:23:27.677619+08:00] [32534] [32535] [server] [ERROR] [llm_daemon.cpp:74] : ERR: Daemon wait pid with 32733 exit, Please check the service log or python log.
2025-07-24 10:23:28,226 [ERROR] model.py:39 - [Model]   >>> Exception:call aclnnArange failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 32742] 2025-07-24-10:23:28.197.611 Parse dynamic kernel config fail.
        TraceBack (most recent call last):
       AclOpKernelInit failed opType
       ArangeAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.

[ERROR] 2025-07-24-10:23:28 (PID:32742, Device:5, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
    return self.python_model.initialize(config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
    self.generator = Generator(
                     ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 79, in __init__
    self.generator_backend = get_generator_backend(model_config)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
    return generator_cls(model_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 96, in __init__
    super().__init__(model_config)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 107, in __init__
    self.model_wrapper = get_model_wrapper(model_config, backend_type)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
    return wrapper_cls(**model_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 46, in __init__
    self.model_runner.load_weights()
  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 180, in load_weights
    self.model = self.model_cls(self.config,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 79, in __init__
    super().__init__(config, weights, **kwargs)
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_multimodal.py", line 184, in __init__
    self.init_vit()
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 88, in init_vit
    self.vision_tower = Qwen25VisionTransformerPretrainedModelATB(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 769, in __init__
    self.rotary_pos_emb = VisionRotaryEmbedding(
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_vl_vit_atb.py", line 70, in __init__
    t = torch.arange(seqlen, device=device, dtype=torch.float)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnArange failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 32742] 2025-07-24-10:23:28.197.611 Parse dynamic kernel config fail.
        TraceBack (most recent call last):
       AclOpKernelInit failed opType
       ArangeAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.

[ERROR] 2025-07-24-10:23:28 (PID:32742, Device:5, RankID:-1) ERR01100 OPS call acl api failed
2025-07-24 10:23:28,315 [ERROR] model.py:42 - [Model]   >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-07-24 10:23:28,395] [32735] [281458429915488] [llmmodels] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-07-24 10:23:28.425558+08:00] [32534] [32534] [server] [WARN] [llm_daemon.cpp:64] : [Daemon] received exit signal[17]
[2025-07-24 10:23:28.425692+08:00] [32534] [32534] [server] [INFO] [llm_daemon.cpp:69] : Daemon wait pid with 32742, status 9
[2025-07-24 10:23:28.425703+08:00] [32534] [32534] [server] [ERROR] [llm_daemon.cpp:74] : ERR: Daemon wait pid with 32742 exit, Please check the service log or python log.
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-4:
Traceback (most recent call last):
  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
    conn = self._tls.connection
           ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
    raise exp
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
    resource_proxy[SUB_PROCESS_STATE].append(True)
  File "<string>", line 2, in append
  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
    self._connect()
  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
    c = SocketClient(address)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
Process ForkServerProcess-3:
Traceback (most recent call last):
  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
    conn = self._tls.connection
           ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

请问是什么问题,有什么解决的方法

我要发帖子