关键报错
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. ... EZ9999: [PID: 697] 2025-08-27-02:53:11.844.134 Op Conv3D does not has any binary. ... launch failed for Conv3D, errno:561000.
完整报错
INFO: 172.17.0.1:44178 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 08-27 03:13:08 [loggers.py:122] Engine 000: Avg prompt throughput: 11.0 tokens/s, Avg generation throughput: 4.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 08-27 03:13:18 [loggers.py:122] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 08-27 03:13:30 [logger.py:41] Received request chatcmpl-a92c737a39bd416b83d096502ee6ad40: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_bos|><|VIDEO|><|vision_eos|>As an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you? Output the thinking process in and final emotion in tags.<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32699, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. Unused or unrecognized kwargs: images. INFO 08-27 03:13:31 [async_llm.py:269] Added request chatcmpl-a92c737a39bd416b83d096502ee6ad40. [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:88:Feature is not supportted and the possible cause is that driver and firmware packages do not match. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] WorkerProc hit an exception. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:31 (PID:5419, Device:1, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: [PID: 5419] 2025-08-27-03:13:31.870.242 Op Conv3D does not has any binary. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:31 (PID:5419, Device:1, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: [PID: 5419] 2025-08-27-03:13:31.870.242 Op Conv3D does not has any binary. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:88:Feature is not supportted and the possible cause is that driver and firmware packages do not match. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] WorkerProc hit an exception. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ERROR 08-27 03:13:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.0) with config: model='/model/Qwen/Qwen2___5-Omni-7B', speculative_config=None, tokenizer='/model/Qwen/Qwen2___5-Omni-7B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Omni-7B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, ERROR 08-27 03:13:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-a92c737a39bd416b83d096502ee6ad40,prompt_token_ids_len=4852,mm_inputs=[{'video_grid_thw': tensor([[16, 26, 46]]), 'pixel_values_videos': tensor([[ 1.2588, 1.2588, 1.2588, ..., 0.9512, 0.9512, 0.9512], ERROR 08-27 03:13:32 [dump_input.py:76] [ 1.2881, 1.2881, 1.3027, ..., 0.9800, 0.9800, 0.9800], ERROR 08-27 03:13:32 [dump_input.py:76] [ 1.2881, 1.2881, 1.2881, ..., 0.8091, 0.8091, 0.8232], ERROR 08-27 03:13:32 [dump_input.py:76] ..., ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7773, -1.7773, -1.7773, ..., -1.4805, -1.4805, -1.4805], ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7627, -1.7627, -1.7627, ..., -1.4658, -1.4658, -1.4658], ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7920, -1.7920, -1.7920, ..., -1.4658, -1.4658, -1.4658]], ERROR 08-27 03:13:32 [dump_input.py:76] dtype=torch.float16)}],mm_hashes=['d434404754ba455dd0c102a9f4d480aacae57efb0c697080892712cf7b2adf44'],mm_positions=[PlaceholderRange(offset=15, length=4784, is_embed=None)],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32699, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={chatcmpl-a92c737a39bd416b83d096502ee6ad40: 2048}, total_num_scheduled_tokens=2048, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={chatcmpl-a92c737a39bd416b83d096502ee6ad40: [0]}, num_common_prefix_blocks=[16], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null) ERROR 08-27 03:13:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.0077167498865183815, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=4852, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0) ERROR 08-27 03:13:32 [core.py:634] EngineCore encountered a fatal error. ERROR 08-27 03:13:32 [core.py:634] Traceback (most recent call last): ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 625, in run_engine_core ERROR 08-27 03:13:32 [core.py:634] engine_core.run_busy_loop() ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 652, in run_busy_loop ERROR 08-27 03:13:32 [core.py:634] self._process_engine_step() ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 677, in _process_engine_step ERROR 08-27 03:13:32 [core.py:634] outputs, model_executed = self.step_fn() ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 267, in step ERROR 08-27 03:13:32 [core.py:634] model_output = self.execute_model_with_error_logging( ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging ERROR 08-27 03:13:32 [core.py:634] raise err ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging ERROR 08-27 03:13:32 [core.py:634] return model_fn(scheduler_output) ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 167, in execute_model ERROR 08-27 03:13:32 [core.py:634] (output, ) = self.collective_rpc( ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 237, in collective_rpc ERROR 08-27 03:13:32 [core.py:634] result = get_response(w, dequeue_timeout) ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 224, in get_response ERROR 08-27 03:13:32 [core.py:634] raise RuntimeError( ERROR 08-27 03:13:32 [core.py:634] RuntimeError: Worker failed with error 'The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. ERROR 08-27 03:13:32 [core.py:634] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. ERROR 08-27 03:13:32 [core.py:634] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. ERROR 08-27 03:13:32 [core.py:634] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. ERROR 08-27 03:13:32 [core.py:634] EZ9999: Inner Error! ERROR 08-27 03:13:32 [core.py:634] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. ERROR 08-27 03:13:32 [core.py:634] TraceBack (most recent call last): ERROR 08-27 03:13:32 [core.py:634] Kernel Run failed. opType: 59, Conv3D ERROR 08-27 03:13:32 [core.py:634] launch failed for Conv3D, errno:561000. ERROR 08-27 03:13:32 [core.py:634] ', please check the stack trace above for the root cause ERROR 08-27 03:13:32 [async_llm.py:416] AsyncLLM output_handler failed. ERROR 08-27 03:13:32 [async_llm.py:416] Traceback (most recent call last): ERROR 08-27 03:13:32 [async_llm.py:416] File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 375, in output_handler ERROR 08-27 03:13:32 [async_llm.py:416] outputs = await engine_core.get_output_async() ERROR 08-27 03:13:32 [async_llm.py:416] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [async_llm.py:416] File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 751, in get_output_async ERROR 08-27 03:13:32 [async_llm.py:416] raise self._format_exception(outputs) from None ERROR 08-27 03:13:32 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. INFO 08-27 03:13:32 [async_llm.py:342] Request chatcmpl-a92c737a39bd416b83d096502ee6ad40 failed (engine dead). INFO: 172.17.0.1:44198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [5012] root@5ee3d66f51a2:/workspace# /usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '
点赞
收藏
回复
分享
举报
已解决
关键报错
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. ... EZ9999: [PID: 697] 2025-08-27-02:53:11.844.134 Op Conv3D does not has any binary. ... launch failed for Conv3D, errno:561000.
完整报错
INFO: 172.17.0.1:44178 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 08-27 03:13:08 [loggers.py:122] Engine 000: Avg prompt throughput: 11.0 tokens/s, Avg generation throughput: 4.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 08-27 03:13:18 [loggers.py:122] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 08-27 03:13:30 [logger.py:41] Received request chatcmpl-a92c737a39bd416b83d096502ee6ad40: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_bos|><|VIDEO|><|vision_eos|>As an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you? Output the thinking process in and final emotion in tags.<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32699, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. Unused or unrecognized kwargs: images. INFO 08-27 03:13:31 [async_llm.py:269] Added request chatcmpl-a92c737a39bd416b83d096502ee6ad40. [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:88:Feature is not supportted and the possible cause is that driver and firmware packages do not match. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] WorkerProc hit an exception. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:31 (PID:5419, Device:1, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: [PID: 5419] 2025-08-27-03:13:31.870.242 Op Conv3D does not has any binary. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:31 (PID:5419, Device:1, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] EZ9999: [PID: 5419] 2025-08-27-03:13:31.870.242 Op Conv3D does not has any binary. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] (VllmWorker rank=1 pid=5419) ERROR 08-27 03:13:31 [multiproc_executor.py:546] [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:88:Feature is not supportted and the possible cause is that driver and firmware packages do not match. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] WorkerProc hit an exception. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Traceback (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 201, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return func(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1604, in execute_model (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] finished_recving) = (self._process_reqs(scheduler_output, (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1220, in _process_reqs (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] self._execute_mm_encoder(scheduler_output) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 950, in _execute_mm_encoder (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] curr_group_outputs = self.model.get_multimodal_embeddings( (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 821, in get_multimodal_embeddings (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeddings = self._process_video_input(multimodal_input) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 705, in _process_video_input (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return self._call_impl(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] return forward_call(*args, **kwargs) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 666, in forward (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] t, h, w = int(t), int(h), int(w) (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ^^^^^^ (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: Inner Error! (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] TraceBack (most recent call last): (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] Kernel Run failed. opType: 59, Conv3D (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] launch failed for Conv3D, errno:561000. (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] (VllmWorker rank=0 pid=5418) ERROR 08-27 03:13:32 [multiproc_executor.py:546] ERROR 08-27 03:13:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.0) with config: model='/model/Qwen/Qwen2___5-Omni-7B', speculative_config=None, tokenizer='/model/Qwen/Qwen2___5-Omni-7B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Omni-7B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, ERROR 08-27 03:13:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-a92c737a39bd416b83d096502ee6ad40,prompt_token_ids_len=4852,mm_inputs=[{'video_grid_thw': tensor([[16, 26, 46]]), 'pixel_values_videos': tensor([[ 1.2588, 1.2588, 1.2588, ..., 0.9512, 0.9512, 0.9512], ERROR 08-27 03:13:32 [dump_input.py:76] [ 1.2881, 1.2881, 1.3027, ..., 0.9800, 0.9800, 0.9800], ERROR 08-27 03:13:32 [dump_input.py:76] [ 1.2881, 1.2881, 1.2881, ..., 0.8091, 0.8091, 0.8232], ERROR 08-27 03:13:32 [dump_input.py:76] ..., ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7773, -1.7773, -1.7773, ..., -1.4805, -1.4805, -1.4805], ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7627, -1.7627, -1.7627, ..., -1.4658, -1.4658, -1.4658], ERROR 08-27 03:13:32 [dump_input.py:76] [-1.7920, -1.7920, -1.7920, ..., -1.4658, -1.4658, -1.4658]], ERROR 08-27 03:13:32 [dump_input.py:76] dtype=torch.float16)}],mm_hashes=['d434404754ba455dd0c102a9f4d480aacae57efb0c697080892712cf7b2adf44'],mm_positions=[PlaceholderRange(offset=15, length=4784, is_embed=None)],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32699, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={chatcmpl-a92c737a39bd416b83d096502ee6ad40: 2048}, total_num_scheduled_tokens=2048, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={chatcmpl-a92c737a39bd416b83d096502ee6ad40: [0]}, num_common_prefix_blocks=[16], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null) ERROR 08-27 03:13:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.0077167498865183815, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=4852, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0) ERROR 08-27 03:13:32 [core.py:634] EngineCore encountered a fatal error. ERROR 08-27 03:13:32 [core.py:634] Traceback (most recent call last): ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 625, in run_engine_core ERROR 08-27 03:13:32 [core.py:634] engine_core.run_busy_loop() ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 652, in run_busy_loop ERROR 08-27 03:13:32 [core.py:634] self._process_engine_step() ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 677, in _process_engine_step ERROR 08-27 03:13:32 [core.py:634] outputs, model_executed = self.step_fn() ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 267, in step ERROR 08-27 03:13:32 [core.py:634] model_output = self.execute_model_with_error_logging( ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging ERROR 08-27 03:13:32 [core.py:634] raise err ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging ERROR 08-27 03:13:32 [core.py:634] return model_fn(scheduler_output) ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 167, in execute_model ERROR 08-27 03:13:32 [core.py:634] (output, ) = self.collective_rpc( ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 237, in collective_rpc ERROR 08-27 03:13:32 [core.py:634] result = get_response(w, dequeue_timeout) ERROR 08-27 03:13:32 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [core.py:634] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 224, in get_response ERROR 08-27 03:13:32 [core.py:634] raise RuntimeError( ERROR 08-27 03:13:32 [core.py:634] RuntimeError: Worker failed with error 'The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnConvolution. ERROR 08-27 03:13:32 [core.py:634] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. ERROR 08-27 03:13:32 [core.py:634] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. ERROR 08-27 03:13:32 [core.py:634] [ERROR] 2025-08-27-03:13:32 (PID:5418, Device:0, RankID:-1) ERR00100 PTA call acl api failed. ERROR 08-27 03:13:32 [core.py:634] EZ9999: Inner Error! ERROR 08-27 03:13:32 [core.py:634] EZ9999: [PID: 5418] 2025-08-27-03:13:32.278.921 Op Conv3D does not has any binary. ERROR 08-27 03:13:32 [core.py:634] TraceBack (most recent call last): ERROR 08-27 03:13:32 [core.py:634] Kernel Run failed. opType: 59, Conv3D ERROR 08-27 03:13:32 [core.py:634] launch failed for Conv3D, errno:561000. ERROR 08-27 03:13:32 [core.py:634] ', please check the stack trace above for the root cause ERROR 08-27 03:13:32 [async_llm.py:416] AsyncLLM output_handler failed. ERROR 08-27 03:13:32 [async_llm.py:416] Traceback (most recent call last): ERROR 08-27 03:13:32 [async_llm.py:416] File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 375, in output_handler ERROR 08-27 03:13:32 [async_llm.py:416] outputs = await engine_core.get_output_async() ERROR 08-27 03:13:32 [async_llm.py:416] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 08-27 03:13:32 [async_llm.py:416] File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 751, in get_output_async ERROR 08-27 03:13:32 [async_llm.py:416] raise self._format_exception(outputs) from None ERROR 08-27 03:13:32 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. INFO 08-27 03:13:32 [async_llm.py:342] Request chatcmpl-a92c737a39bd416b83d096502ee6ad40 failed (engine dead). INFO: 172.17.0.1:44198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [5012] root@5ee3d66f51a2:/workspace# /usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '