The old environment variable ATB_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31.
Please use the new environment variable MINDIE_LOG_LEVEL as soon as
possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon
as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be
deprecated on 2025/12/31. Please use the new environment variable
MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated
on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as
soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated
on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be
deprecated on 2025/12/31. Please use the new environment variable
MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as
soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31.
Please use the new environment variable MINDIE_LOG_LEVEL as soon as
possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
You are using a model of type qwen2_5_vl to instantiate a model of type .
This is not supported for all configurations of models and can yield errors.
Using a slow image processor as `use_fast` is unset and a slow processor was
saved with this model. `use_fast=True` will be the default behavior in
v4.52, even if the model was saved with a slow processor. This will result
in minor differences in outputs. You'll still be able to use a slow
processor with `use_fast=False`.
You are using a model of type qwen2_5_vl to instantiate a model of type .
This is not supported for all configurations of models and can yield errors.
[2025-07-08 17:04:57.178431] [error] [1256320] [linear_operation.cpp:629]
LinearOperation Error code: 8. value of inTensor0 k and value of inTensor1 k
should be equal, inTensor0 k = 3424, inTensor1 k = 3420. Please check the
shape of inTensors.
[2025-07-08 17:04:57.178503] [error] [1256320] [operation_base.cpp:267]
LinearOperation infer shape check fail, error code: 8
[2025-07-08 17:04:57.178515] [error] [1256320] [graph_operation.cpp:428]
Qwen25VL_VIT_layer_0_graph node[12] infer shape fail, error code: 8
[2025-07-08 17:04:57.178566] [error] [1256320] [operation_base.cpp:273]
Qwen25VL_VIT_layer_0_graph infer shape fail, error code: 8
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/models/qwen2_vl/run_pa.py",
line 343, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/models/qwen2_vl/run_pa.py",
line 141, in warm_up
generate_req([single_req], self.model, self.max_batch_size,
self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line
1143, in generate_req
generate_token_with_clocking(model, cache_manager, batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 810,
in generate_token_with_clocking
res = generate_token(model, cache_manager, input_batch_in)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 587,
in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line
297, in forward
res = self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.
py", line 118, in forward
inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts =
self.prepare_prefill_token_service(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.
py", line 197, in prepare_prefill_token_service
image_features =
self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vi
t_atb.py", line 867, in forward
vision_features = self.encoder(
^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vi
t_atb.py", line 714, in forward
hidden_states = self.graph.forward(self.graph_inputs,
self.graph_outputs, self.graph_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Qwen25VL_VIT_graph nodes[1] infershape fail, enable log:
export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1
[ERROR] 2025-07-08-17:05:02 (PID:1256320, Device:0, RankID:-1) ERR99999
UNKNOWN application exception
点赞
收藏
回复
分享
举报
已解决
The old environment variable ATB_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31.
Please use the new environment variable MINDIE_LOG_LEVEL as soon as
possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon
as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be
deprecated on 2025/12/31. Please use the new environment variable
MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated
on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as
soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated
on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as
soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be
deprecated on 2025/12/31. Please use the new environment variable
MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as
soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31.
Please use the new environment variable MINDIE_LOG_LEVEL as soon as
possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on
2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as
soon as possible.
You are using a model of type qwen2_5_vl to instantiate a model of type .
This is not supported for all configurations of models and can yield errors.
Using a slow image processor as `use_fast` is unset and a slow processor was
saved with this model. `use_fast=True` will be the default behavior in
v4.52, even if the model was saved with a slow processor. This will result
in minor differences in outputs. You'll still be able to use a slow
processor with `use_fast=False`.
You are using a model of type qwen2_5_vl to instantiate a model of type .
This is not supported for all configurations of models and can yield errors.
[2025-07-08 17:04:57.178431] [error] [1256320] [linear_operation.cpp:629]
LinearOperation Error code: 8. value of inTensor0 k and value of inTensor1 k
should be equal, inTensor0 k = 3424, inTensor1 k = 3420. Please check the
shape of inTensors.
[2025-07-08 17:04:57.178503] [error] [1256320] [operation_base.cpp:267]
LinearOperation infer shape check fail, error code: 8
[2025-07-08 17:04:57.178515] [error] [1256320] [graph_operation.cpp:428]
Qwen25VL_VIT_layer_0_graph node[12] infer shape fail, error code: 8
[2025-07-08 17:04:57.178566] [error] [1256320] [operation_base.cpp:273]
Qwen25VL_VIT_layer_0_graph infer shape fail, error code: 8
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/models/qwen2_vl/run_pa.py",
line 343, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/models/qwen2_vl/run_pa.py",
line 141, in warm_up
generate_req([single_req], self.model, self.max_batch_size,
self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line
1143, in generate_req
generate_token_with_clocking(model, cache_manager, batch)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 810,
in generate_token_with_clocking
res = generate_token(model, cache_manager, input_batch_in)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 587,
in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line
297, in forward
res = self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.
py", line 118, in forward
inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts =
self.prepare_prefill_token_service(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.
py", line 197, in prepare_prefill_token_service
image_features =
self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vi
t_atb.py", line 867, in forward
vision_features = self.encoder(
^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line
1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vi
t_atb.py", line 714, in forward
hidden_states = self.graph.forward(self.graph_inputs,
self.graph_outputs, self.graph_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Qwen25VL_VIT_graph nodes[1] infershape fail, enable log:
export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1
[ERROR] 2025-07-08-17:05:02 (PID:1256320, Device:0, RankID:-1) ERR99999
UNKNOWN application exception