INFO 08-12 01:17:01 [__init__.py:39] Available plugins for group vllm.platform_plugins:
INFO 08-12 01:17:01 [__init__.py:41] - ascend -> vllm_ascend:register
INFO 08-12 01:17:01 [__init__.py:44] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-12 01:17:01 [__init__.py:235] Platform plugin ascend is activated
WARNING 08-12 01:17:02 [_custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
INFO 08-12 01:17:05 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B
2025-08-12 01:17:09,246 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B
2025-08-12 01:17:11,712 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 08-12 01:17:24 [config.py:841] This model supports multiple tasks: {'reward', 'embed', 'classify', 'generate'}. Defaulting to 'generate'.
INFO 08-12 01:17:24 [config.py:1472] Using max model len 40960
INFO 08-12 01:17:24 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-12 01:17:24 [platform.py:174] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 08-12 01:17:24 [utils.py:321] Calculated maximum supported batch sizes for ACL graph: 66
INFO 08-12 01:17:24 [utils.py:336] Adjusted ACL graph batch sizes for Qwen3ForCausalLM model (layers: 28): 67 → 66 sizes
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B
2025-08-12 01:17:28,842 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B
2025-08-12 01:17:32,924 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 08-12 01:17:32 [core.py:526] Waiting for init message from front-end.
INFO 08-12 01:17:32 [core.py:69] Initializing a V1 LLM engine (v0.9.2) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-0.6B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.unified_ascend_attention_with_output"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
[rank0]:[W812 01:17:43.665372666 ProcessGroupGloo.cpp:715] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
INFO 08-12 01:17:54 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 08-12 01:18:01 [model_runner_v1.py:1745] Starting to load model Qwen/Qwen3-0.6B...
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B
2025-08-12 01:18:06,622 - modelscope - INFO - Got 1 files, start to download ...
Downloading [model.safetensors]: 100%|█████████████████████████████████████████| 1.40G/1.40G [12:46<00:00, 1.96MB/s]
Processing 1 items: 100%|█████████████████████████████████████████████████████████| 1.00/1.00 [12:46<00:00, 767s/it]
2025-08-12 01:30:53,281 - modelscope - INFO - Download model 'Qwen/Qwen3-0.6B' successfully.
2025-08-12 01:30:53,282 - modelscope - INFO - Target directory already exists, skipping creation.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.99it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.99it/s]
INFO 08-12 01:30:53 [default_loader.py:272] Loading weights took 0.34 seconds
INFO 08-12 01:30:54 [model_runner_v1.py:1777] Loading model weights took 1.1202 GB
INFO 08-12 01:31:08 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/c878ede1af/rank_0_0/backbone for vLLM's torch.compile
INFO 08-12 01:31:08 [backends.py:519] Dynamo bytecode transform time: 7.62 s
INFO 08-12 01:31:11 [backends.py:193] Compiling a graph for general shape takes 1.66 s
.INFO 08-12 01:31:18 [monitor.py:34] torch.compile takes 9.28 s in total
INFO 08-12 01:31:19 [worker_v1.py:181] Available memory: 16910831104, total memory: 22573076480
INFO 08-12 01:31:19 [kv_cache_utils.py:716] GPU KV cache size: 147,328 tokens
INFO 08-12 01:31:19 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 3.60x
[WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:161:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
ERROR 08-12 01:31:20 [core.py:586] EngineCore failed to start.
ERROR 08-12 01:31:20 [core.py:586] Traceback (most recent call last):
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
ERROR 08-12 01:31:20 [core.py:586] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
ERROR 08-12 01:31:20 [core.py:586] super().__init__(vllm_config, executor_class, log_stats,
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 82, in __init__
ERROR 08-12 01:31:20 [core.py:586] self._initialize_kv_caches(vllm_config)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 169, in _initialize_kv_caches
ERROR 08-12 01:31:20 [core.py:586] self.model_executor.initialize_from_config(kv_cache_configs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
ERROR 08-12 01:31:20 [core.py:586] self.collective_rpc("compile_or_warm_up_model")
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 08-12 01:31:20 [core.py:586] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 2736, in run_method
ERROR 08-12 01:31:20 [core.py:586] return func(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 253, in compile_or_warm_up_model
ERROR 08-12 01:31:20 [core.py:586] self.model_runner.capture_model()
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2064, in capture_model
ERROR 08-12 01:31:20 [core.py:586] self._dummy_run(num_tokens)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 08-12 01:31:20 [core.py:586] return func(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1663, in _dummy_run
ERROR 08-12 01:31:20 [core.py:586] hidden_states = model(
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 302, in forward
ERROR 08-12 01:31:20 [core.py:586] hidden_states = self.model(input_ids, positions, intermediate_tensors,
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 246, in __call__
ERROR 08-12 01:31:20 [core.py:586] model_output = self.forward(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 337, in forward
ERROR 08-12 01:31:20 [core.py:586] def forward(
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
ERROR 08-12 01:31:20 [core.py:586] return fn(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped
ERROR 08-12 01:31:20 [core.py:586] return self._wrapped_call(self, *args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 361, in __call__
ERROR 08-12 01:31:20 [core.py:586] raise e
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 348, in __call__
ERROR 08-12 01:31:20 [core.py:586] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs)
ERROR 08-12 01:31:20 [core.py:586] File "<eval_with_key>.58", line 234, in forward
ERROR 08-12 01:31:20 [core.py:586] submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None
ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
ERROR 08-12 01:31:20 [core.py:586] with torch.npu.graph(aclgraph, pool=self.graph_pool):
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
ERROR 08-12 01:31:20 [core.py:586] self.npu_graph.capture_begin(
ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
ERROR 08-12 01:31:20 [core.py:586] super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
ERROR 08-12 01:31:20 [core.py:586] RuntimeError: status == aclmdlRICaptureStatus::ACL_MODEL_RI_CAPTURE_STATUS_ACTIVE INTERNAL ASSERT FAILED at "build/CMakeFiles/torch_npu.dir/compiler_depend.ts":162, please report a bug to PyTorch.
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 590, in run_engine_core
raise e
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 82, in __init__
self._initialize_kv_caches(vllm_config)
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 169, in _initialize_kv_caches
self.model_executor.initialize_from_config(kv_cache_configs)
File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
self.collective_rpc("compile_or_warm_up_model")
File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 2736, in run_method
return func(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 253, in compile_or_warm_up_model
self.model_runner.capture_model()
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2064, in capture_model
self._dummy_run(num_tokens)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1663, in _dummy_run
hidden_states = model(
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 302, in forward
hidden_states = self.model(input_ids, positions, intermediate_tensors,
File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 246, in __call__
model_output = self.forward(*args, **kwargs)
File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 337, in forward
def forward(
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
return fn(*args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped
return self._wrapped_call(self, *args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 361, in __call__
raise e
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 348, in __call__
return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "<eval_with_key>.58", line 234, in forward
submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None
File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
with torch.npu.graph(aclgraph, pool=self.graph_pool):
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
self.npu_graph.capture_begin(
File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
RuntimeError: status == aclmdlRICaptureStatus::ACL_MODEL_RI_CAPTURE_STATUS_ACTIVE INTERNAL ASSERT FAILED at "build/CMakeFiles/torch_npu.dir/compiler_depend.ts":162, please report a bug to PyTorch.
Traceback (most recent call last):
File "/workspace/test.py", line 9, in <module>
llm = LLM(model="Qwen/Qwen3-0.6B")
File "/vllm-workspace/vllm/vllm/entrypoints/llm.py", line 271, in __init__
self.llm_engine = LLMEngine.from_engine_args(
File "/vllm-workspace/vllm/vllm/engine/llm_engine.py", line 501, in from_engine_args
return engine_cls.from_vllm_config(
File "/vllm-workspace/vllm/vllm/v1/engine/llm_engine.py", line 124, in from_vllm_config
return cls(vllm_config=vllm_config,
File "/vllm-workspace/vllm/vllm/v1/engine/llm_engine.py", line 101, in __init__
self.engine_core = EngineCoreClient.make_client(
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 75, in make_client
return SyncMPClient(vllm_config, executor_class, log_stats)
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 503, in __init__
super().__init__(
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 403, in __init__
with launch_core_engines(vllm_config, executor_class,
File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 434, in launch_core_engines
wait_for_engine_startup(
File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
[ERROR] 2025-08-12-01:31:20 (PID:909, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
点赞
收藏
回复
分享
举报
已解决
屏幕日志如下：
INFO 08-12 01:17:01 [__init__.py:39] Available plugins for group vllm.platform_plugins: INFO 08-12 01:17:01 [__init__.py:41] - ascend -> vllm_ascend:register INFO 08-12 01:17:01 [__init__.py:44] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. INFO 08-12 01:17:01 [__init__.py:235] Platform plugin ascend is activated WARNING 08-12 01:17:02 [_custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") INFO 08-12 01:17:05 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available. WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP. WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration. WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration. WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM. WARNING 08-12 01:17:05 [registry.py:413] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM. WARNING 08-12 01:17:05 [registry.py:413] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM. Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B 2025-08-12 01:17:09,246 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B 2025-08-12 01:17:11,712 - modelscope - INFO - Target directory already exists, skipping creation. INFO 08-12 01:17:24 [config.py:841] This model supports multiple tasks: {'reward', 'embed', 'classify', 'generate'}. Defaulting to 'generate'. INFO 08-12 01:17:24 [config.py:1472] Using max model len 40960 INFO 08-12 01:17:24 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=8192. INFO 08-12 01:17:24 [platform.py:174] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode INFO 08-12 01:17:24 [utils.py:321] Calculated maximum supported batch sizes for ACL graph: 66 INFO 08-12 01:17:24 [utils.py:336] Adjusted ACL graph batch sizes for Qwen3ForCausalLM model (layers: 28): 67 → 66 sizes Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B 2025-08-12 01:17:28,842 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B 2025-08-12 01:17:32,924 - modelscope - INFO - Target directory already exists, skipping creation. INFO 08-12 01:17:32 [core.py:526] Waiting for init message from front-end. INFO 08-12 01:17:32 [core.py:69] Initializing a V1 LLM engine (v0.9.2) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-0.6B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.unified_ascend_attention_with_output"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null} [rank0]:[W812 01:17:43.665372666 ProcessGroupGloo.cpp:715] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator()) INFO 08-12 01:17:54 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 INFO 08-12 01:18:01 [model_runner_v1.py:1745] Starting to load model Qwen/Qwen3-0.6B... Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B 2025-08-12 01:18:06,622 - modelscope - INFO - Got 1 files, start to download ... Downloading [model.safetensors]: 100%|█████████████████████████████████████████| 1.40G/1.40G [12:46<00:00, 1.96MB/s] Processing 1 items: 100%|█████████████████████████████████████████████████████████| 1.00/1.00 [12:46<00:00, 767s/it] 2025-08-12 01:30:53,281 - modelscope - INFO - Download model 'Qwen/Qwen3-0.6B' successfully. 2025-08-12 01:30:53,282 - modelscope - INFO - Target directory already exists, skipping creation. Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.99it/s] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.99it/s] INFO 08-12 01:30:53 [default_loader.py:272] Loading weights took 0.34 seconds INFO 08-12 01:30:54 [model_runner_v1.py:1777] Loading model weights took 1.1202 GB INFO 08-12 01:31:08 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/c878ede1af/rank_0_0/backbone for vLLM's torch.compile INFO 08-12 01:31:08 [backends.py:519] Dynamo bytecode transform time: 7.62 s INFO 08-12 01:31:11 [backends.py:193] Compiling a graph for general shape takes 1.66 s .INFO 08-12 01:31:18 [monitor.py:34] torch.compile takes 9.28 s in total INFO 08-12 01:31:19 [worker_v1.py:181] Available memory: 16910831104, total memory: 22573076480 INFO 08-12 01:31:19 [kv_cache_utils.py:716] GPU KV cache size: 147,328 tokens INFO 08-12 01:31:19 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 3.60x [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match. [WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:161:Feature is not supportted and the possible cause is that driver and firmware packages do not match. ERROR 08-12 01:31:20 [core.py:586] EngineCore failed to start. ERROR 08-12 01:31:20 [core.py:586] Traceback (most recent call last): ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core ERROR 08-12 01:31:20 [core.py:586] engine_core = EngineCoreProc(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__ ERROR 08-12 01:31:20 [core.py:586] super().__init__(vllm_config, executor_class, log_stats, ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 82, in __init__ ERROR 08-12 01:31:20 [core.py:586] self._initialize_kv_caches(vllm_config) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 169, in _initialize_kv_caches ERROR 08-12 01:31:20 [core.py:586] self.model_executor.initialize_from_config(kv_cache_configs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config ERROR 08-12 01:31:20 [core.py:586] self.collective_rpc("compile_or_warm_up_model") ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc ERROR 08-12 01:31:20 [core.py:586] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 2736, in run_method ERROR 08-12 01:31:20 [core.py:586] return func(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 253, in compile_or_warm_up_model ERROR 08-12 01:31:20 [core.py:586] self.model_runner.capture_model() ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2064, in capture_model ERROR 08-12 01:31:20 [core.py:586] self._dummy_run(num_tokens) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 08-12 01:31:20 [core.py:586] return func(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1663, in _dummy_run ERROR 08-12 01:31:20 [core.py:586] hidden_states = model( ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 302, in forward ERROR 08-12 01:31:20 [core.py:586] hidden_states = self.model(input_ids, positions, intermediate_tensors, ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 246, in __call__ ERROR 08-12 01:31:20 [core.py:586] model_output = self.forward(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 337, in forward ERROR 08-12 01:31:20 [core.py:586] def forward( ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn ERROR 08-12 01:31:20 [core.py:586] return fn(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped ERROR 08-12 01:31:20 [core.py:586] return self._wrapped_call(self, *args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 361, in __call__ ERROR 08-12 01:31:20 [core.py:586] raise e ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 348, in __call__ ERROR 08-12 01:31:20 [core.py:586] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl ERROR 08-12 01:31:20 [core.py:586] return self._call_impl(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl ERROR 08-12 01:31:20 [core.py:586] return forward_call(*args, **kwargs) ERROR 08-12 01:31:20 [core.py:586] File "<eval_with_key>.58", line 234, in forward ERROR 08-12 01:31:20 [core.py:586] submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None ERROR 08-12 01:31:20 [core.py:586] File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__ ERROR 08-12 01:31:20 [core.py:586] with torch.npu.graph(aclgraph, pool=self.graph_pool): ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__ ERROR 08-12 01:31:20 [core.py:586] self.npu_graph.capture_begin( ERROR 08-12 01:31:20 [core.py:586] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin ERROR 08-12 01:31:20 [core.py:586] super().capture_begin(pool=pool, capture_error_mode=capture_error_mode) ERROR 08-12 01:31:20 [core.py:586] RuntimeError: status == aclmdlRICaptureStatus::ACL_MODEL_RI_CAPTURE_STATUS_ACTIVE INTERNAL ASSERT FAILED at "build/CMakeFiles/torch_npu.dir/compiler_depend.ts":162, please report a bug to PyTorch. Process EngineCore_0: Traceback (most recent call last): File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 590, in run_engine_core raise e File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core engine_core = EngineCoreProc(*args, **kwargs) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__ super().__init__(vllm_config, executor_class, log_stats, File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 82, in __init__ self._initialize_kv_caches(vllm_config) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 169, in _initialize_kv_caches self.model_executor.initialize_from_config(kv_cache_configs) File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config self.collective_rpc("compile_or_warm_up_model") File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc answer = run_method(self.driver_worker, method, args, kwargs) File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 2736, in run_method return func(*args, **kwargs) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 253, in compile_or_warm_up_model self.model_runner.capture_model() File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2064, in capture_model self._dummy_run(num_tokens) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1663, in _dummy_run hidden_states = model( File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 302, in forward hidden_states = self.model(input_ids, positions, intermediate_tensors, File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 246, in __call__ model_output = self.forward(*args, **kwargs) File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 337, in forward def forward( File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn return fn(*args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped return self._wrapped_call(self, *args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 361, in __call__ raise e File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/fx/graph_module.py", line 348, in __call__ return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) File "<eval_with_key>.58", line 234, in forward submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__ with torch.npu.graph(aclgraph, pool=self.graph_pool): File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__ self.npu_graph.capture_begin( File "/usr/local/python3.10.17/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin super().capture_begin(pool=pool, capture_error_mode=capture_error_mode) RuntimeError: status == aclmdlRICaptureStatus::ACL_MODEL_RI_CAPTURE_STATUS_ACTIVE INTERNAL ASSERT FAILED at "build/CMakeFiles/torch_npu.dir/compiler_depend.ts":162, please report a bug to PyTorch. Traceback (most recent call last): File "/workspace/test.py", line 9, in <module> llm = LLM(model="Qwen/Qwen3-0.6B") File "/vllm-workspace/vllm/vllm/entrypoints/llm.py", line 271, in __init__ self.llm_engine = LLMEngine.from_engine_args( File "/vllm-workspace/vllm/vllm/engine/llm_engine.py", line 501, in from_engine_args return engine_cls.from_vllm_config( File "/vllm-workspace/vllm/vllm/v1/engine/llm_engine.py", line 124, in from_vllm_config return cls(vllm_config=vllm_config, File "/vllm-workspace/vllm/vllm/v1/engine/llm_engine.py", line 101, in __init__ self.engine_core = EngineCoreClient.make_client( File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 75, in make_client return SyncMPClient(vllm_config, executor_class, log_stats) File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 503, in __init__ super().__init__( File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 403, in __init__ with launch_core_engines(vllm_config, executor_class, File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 434, in launch_core_engines wait_for_engine_startup( File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup raise RuntimeError("Engine core initialization failed. " RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} [ERROR] 2025-08-12-01:31:20 (PID:909, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception系统环境如下：