昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
Mindie service推理服务报错，无法生成回复，{"error":"Failed to get engine response.","error_type":"Incomplete Generation"}
收藏回复举报
Mindie service推理服务报错，无法生成回复，{"error":"Failed to get engine response.","error_type":"Incomplete Generation"}
t('forum.solved') 已解决
新人帖
发表于2025-03-29 11:35:17
0 查看

昇腾910B物理机部署mindie service推理服务，模型是Qwen-1.5-7B，已经成功启动mindie service，如图

cke_2478.png

但是通过http调用服务时无法返回生成内容，显示生成回复失败，如图

cke_10168.png

log中的报错信息如下：

[2025-03-29 10:46:02.715+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:438] : [model_backend] [=========== Backend rank 0, start processing the broadcast message =============]

[2025-03-29 10:46:02.715+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:466] : [model_backend] [=========== Backend rank 0, before DeSerializeExecuteRequests =============]

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:62] : [py receiver] recv a task

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:66] : [python receiver] add a task to infer_queue, task type is:1

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:81] : [python thread: infer] start to process a inference task, task type is:1

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:125] : [Model]      >>>global rank-0 enter execute, batch size:1, config:{'EXECUTE_TYPE': '1'}

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:354] : [Model]      >>> Enter generate()

[2025-03-29 10:46:02.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:557] : [Model]      >>> rank-0 make metadata spend time: 0

                batch size: 1

                reqid is: [0]

[2025-03-29 10:46:02.715+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:483] : [model_backend] [=========== Backend rank 0, after DeSerializeExecuteRequests, before calling executor doTask =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:601] : [Model]      >>> rank-0 model spend time: 437782320

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:370] : [Model]      >>>create response spend time: 0, batch size: 1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:379] : [Model]      >>>global rank-0 exec func spend time: 439,

                        batch size: 1, stage: prefill, is_Mix is False

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:383] : [Model]      >>>global rank-0 total spend time: 439

[2025-03-29 10:46:03.156+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:543] : [model_backend] [=========== Backend rank 0, start processing the recv message =============]

[2025-03-29 10:46:03.156+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:602] : [model_backend] [=========== Backend rank 0, after calling executor InstanceExecute =============]

[2025-03-29 10:46:03.156+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:606] : [model_backend] [=========== Backend rank 0, before SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.156+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:615] : [model_backend] [=========== Backend rank 0, after SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:78] : [python thread: infer] block waiting for a inference task

[2025-03-29 10:46:03.156+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:626] : [model_backend] [=========== Backend rank 0, finish sending infer response message, message len:165 =============]

[2025-03-29 10:46:03.157+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:438] : [model_backend] [=========== Backend rank 0, start processing the broadcast message =============]

[2025-03-29 10:46:03.157+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:466] : [model_backend] [=========== Backend rank 0, before DeSerializeExecuteRequests =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:62] : [py receiver] recv a task

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:66] : [python receiver] add a task to infer_queue, task type is:1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:81] : [python thread: infer] start to process a inference task, task type is:1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:125] : [Model]      >>>global rank-0 enter execute, batch size:1, config:{'EXECUTE_TYPE': '2'}

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:354] : [Model]      >>> Enter generate()

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:557] : [Model]      >>> rank-0 make metadata spend time: 0

                batch size: 1

                reqid is: [0]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:601] : [Model]      >>> rank-0 model spend time: 472829740

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:370] : [Model]      >>>create response spend time: 0, batch size: 1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:379] : [Model]      >>>global rank-0 exec func spend time: 474,

                        batch size: 1, stage: decode, is_Mix is False

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:383] : [Model]      >>>global rank-0 total spend time: 913

[2025-03-29 10:46:03.157+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:483] : [model_backend] [=========== Backend rank 0, after DeSerializeExecuteRequests, before calling executor doTask =============]

[2025-03-29 10:46:03.632+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:543] : [model_backend] [=========== Backend rank 0, start processing the recv message =============]

[2025-03-29 10:46:03.632+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:602] : [model_backend] [=========== Backend rank 0, after calling executor InstanceExecute =============]

[2025-03-29 10:46:03.632+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:606] : [model_backend] [=========== Backend rank 0, before SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.632+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:615] : [model_backend] [=========== Backend rank 0, after SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:78] : [python thread: infer] block waiting for a inference task

[2025-03-29 10:46:03.632+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:626] : [model_backend] [=========== Backend rank 0, finish sending infer response message, message len:165 =============]

[2025-03-29 10:46:03.634+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:438] : [model_backend] [=========== Backend rank 0, start processing the broadcast message =============]

[2025-03-29 10:46:03.634+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:466] : [model_backend] [=========== Backend rank 0, before DeSerializeExecuteRequests =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:62] : [py receiver] recv a task

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:66] : [python receiver] add a task to infer_queue, task type is:1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:81] : [python thread: infer] start to process a inference task, task type is:1

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:125] : [Model]      >>>global rank-0 enter execute, batch size:1, config:{'EXECUTE_TYPE': '2'}

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:354] : [Model]      >>> Enter generate()

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [standard_model.py:557] : [Model]      >>> rank-0 make metadata spend time: 0

                batch size: 1

                reqid is: [0]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [ERROR] [standard_model.py:142] : [Model]      >>>Execute type:2, Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in execute

    return self._decode(requests)

           ^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 231, in _decode

    return self.__generate(requests, is_prefill=False, is_mix=False)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 357, in __generate

    request_tokens_np, eos_np, truncation_indices = self.__handle_requests(requests, is_prefill, is_mix)

                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 580, in __handle_requests

    request_tokens_np, eos_np, _, truncation_indices = self.generator.generate_token(metadata)

                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 190, in generate_token

    raise e

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 163, in generate_token

    self.plugin.generate_token(input_metadata))

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 95, in wrapper

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin.py", line 22, in generate_token

    logits = self.generator_backend.forward(model_inputs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 73, in wrapper

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 170, in forward

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 109, in forward

    logits = self.forward_tensor(

             ^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 136, in forward_tensor

    logits = self.model_runner.forward(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward

    return self.model.forward(**kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 466, in forward

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 434, in execute_ascend_operator

    acl_model_out = model_operation.execute(acl_inputs, acl_param)

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

[2025-03-29 10:46:03.634+08:00] [65140] [65144] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:483] : [model_backend] [=========== Backend rank 0, after DeSerializeExecuteRequests, before calling executor doTask =============]

[2025-03-29 10:46:03.649+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:543] : [model_backend] [=========== Backend rank 0, start processing the recv message =============]

[2025-03-29 10:46:03.649+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:602] : [model_backend] [=========== Backend rank 0, after calling executor InstanceExecute =============]

[2025-03-29 10:46:03.649+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:606] : [model_backend] [=========== Backend rank 0, before SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.649+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:615] : [model_backend] [=========== Backend rank 0, after SerializeInferResponsesMessage =============]

[2025-03-29 10:46:03.029+0800] [65140] [281473525084512] [mindie-server] [DEBUG] [model.py:78] : [python thread: infer] block waiting for a inference task

[2025-03-29 10:46:03.649+08:00] [65140] [65342] [mindie-server] [DEBUG] [slave_IPC_communicator.cpp:626] : [model_backend] [=========== Backend rank 0, finish sending infer response message, message len:13 =============]

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情