[WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:77:Feature is not supported and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:77:Feature is not supported and the possible cause is that driver and firmware packages do not match.
[ERROR] WorkerProc hit an exception
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
output = func(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
[ERROR] 2025-10-17-16:26:11 (PID:696, Device:0, RankID:-1) ERROR0100 PTA call acl api failed.
EZ1001: (PID:696) 2025-10-17-16:26:11.426.903 Assert ((executor) != nullptr) failed
Check NnopbaseRunWithWorkspace(executor, stream, workspace, workspaceSize) failed
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
The op operator name is aclnnMoeGatingTopKSoftmax.
[ERROR] 2025-10-17-16:26:11 (PID:696, Device:0, RankID:-1) ERROR0100 PTA call acl api failed.
EZ1001: (PID:696) 2025-10-17-16:26:11.426.903 Assert ((executor) != nullptr) failed
Check NnopbaseRunWithWorkspace(executor, stream, workspace, workspaceSize) failed
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
The op operator name is aclnnMoeGatingTopKSoftmax.
[ERROR] EngineCore_DP0 (pid=560) ERROR 10-17 16:26:14 [core.py:718] failed to start.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 709, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 505, in __init__
super().__init__(vllm_config, executor_class, log_stats)
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 91, in __init__
self._initialize_kv_caches(vllm_config)
[WARN]operator(),build/CMakeFiles/torch_npu.dir/compiler_depend.ts:77:Feature is not supported and the possible cause is that driver and firmware packages do not match.
[ERROR] WorkerProc hit an exception
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
output = func(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
[ERROR] 2025-10-17-16:26:11 (PID:696, Device:0, RankID:-1) ERROR0100 PTA call acl api failed.
EZ1001: (PID:696) 2025-10-17-16:26:11.426.903 Assert ((executor) != nullptr) failed
Check NnopbaseRunWithWorkspace(executor, stream, workspace, workspaceSize) failed
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
The op operator name is aclnnMoeGatingTopKSoftmax.
[ERROR] 2025-10-17-16:26:11 (PID:696, Device:0, RankID:-1) ERROR0100 PTA call acl api failed.
EZ1001: (PID:696) 2025-10-17-16:26:11.426.903 Assert ((executor) != nullptr) failed
Check NnopbaseRunWithWorkspace(executor, stream, workspace, workspaceSize) failed
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/executor/multiproc_executor.py", line 649, in worker_busy_loop
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 169, in determine_available_memory
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2269, in profile_run
hidden_states = hidden_states[logit_indices]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current worker fails to start.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation.
Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
The op operator name is aclnnMoeGatingTopKSoftmax.
[ERROR] EngineCore_DP0 (pid=560) ERROR 10-17 16:26:14 [core.py:718] failed to start.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 709, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 505, in __init__
super().__init__(vllm_config, executor_class, log_stats)
File "/vllm-workspace/vllm-ascend/vllm_ascend/engine/core.py", line 91, in __init__
self._initialize_kv_caches(vllm_config)