环境信息
服务器架构：arm64
驱动版本：25.5.0
固件版本：7.8.0.5.216
启动镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.2.RC1-800I-A2-aarch64
启动命令：
docker run -u root -e ASCEND_VISIBLE_DEVICES=3 -e ENABLE_BOOST=True -itd --name embedding_srv -p 7712:7712 \
-v /data01/models:/home/HwHiAiUser/model \
swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.2.RC1-800I-A2-aarch64 bge-m3 0.0.0.0 7712
日志信息：
Model '/home/HwHiAiUser/model/bge-m3' exists.
run on device 0
Starting TEI service on 0.0.0.0:7712...
2026-01-09T09:48:13.822648Z INFO text_embeddings_router: router/src/main.rs:189: Args { model_id: "b**-*3", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 64, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hf_token: None, hostname: "0.0.0.0", port: 7712, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: None, payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2026-01-09T09:48:15.098995Z INFO text_embeddings_router: router/src/lib.rs:212: Maximum number of tokens per request: 8192
2026-01-09T09:48:15.099406Z INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 192 tokenization workers
2026-01-09T09:49:54.954894Z INFO text_embeddings_router: router/src/lib.rs:254: Starting model backend
2026-01-09T09:49:54.958993Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:68: Starting Python backend
2026-01-09T09:50:05.149443Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:132: Waiting for Python backend to be ready...
2026-01-09T09:50:09.033160Z INFO python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:37: backend device: npu
2026-01-09T09:50:11.528433Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:129: Python backend ready in 16.38231977s
2026-01-09T09:50:11.529131Z INFO python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:37: Server started at unix:///tmp/text-embeddings-inference-server
2026-01-09T09:50:11.683865Z ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.
Traceback (most recent call last):
File "/usr/local/bin/python-text-embeddings-server", line 7, in <module>
sys.exit(app())
File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
return get_command(self)(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 716, in main
return _main(
File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 216, in _main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
return callback(**use_params) # type: ignore
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
server.serve(model_path, dtype, uds_path, pool)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve
asyncio.run(serve_inner(model_path, dtype))
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
self._run_once()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.11/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
return await self.intercept(
> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept
return await response
File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor
raise error
File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor
return await behavior(request_or_iterator, context)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 39, in Embed
embeddings = self.model.embed(batch)
File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 91, in embed
return self._process_default(batch, kwargs)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 122, in _process_default
output = self.model(**kwargs)
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/Ascend/mxRag/ops/transformer_adapter/modeling_xlm_roberta_adapter.py", line 91, in forward_boost
position_ids = position_ids.view(-1, seq_length).long()
RuntimeError: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.
2026-01-09T09:50:11.684434Z ERROR health:embed:embed: backend_grpc_client: backends/grpc-client/src/lib.rs:25: Server error: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.
2026-01-09T09:50:11.732239Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:146: Python backend process terminated
Error: Model backend is not healthy
Caused by:
Server error: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.
点赞
收藏
回复
分享
举报
已解决
环境信息
服务器架构：arm64
驱动版本：25.5.0
固件版本：7.8.0.5.216
启动镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.2.RC1-800I-A2-aarch64
启动命令：
docker run -u root -e ASCEND_VISIBLE_DEVICES=3 -e ENABLE_BOOST=True -itd --name embedding_srv -p 7712:7712 \
-v /data01/models:/home/HwHiAiUser/model \
swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.2.RC1-800I-A2-aarch64 bge-m3 0.0.0.0 7712
日志信息：
Model '/home/HwHiAiUser/model/bge-m3' exists.
run on device 0
Starting TEI service on 0.0.0.0:7712...
2026-01-09T09:48:13.822648Z INFO text_embeddings_router: router/src/main.rs:189: Args { model_id: "b**-*3", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 64, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hf_token: None, hostname: "0.0.0.0", port: 7712, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: None, payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2026-01-09T09:48:15.098995Z INFO text_embeddings_router: router/src/lib.rs:212: Maximum number of tokens per request: 8192
2026-01-09T09:48:15.099406Z INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 192 tokenization workers
2026-01-09T09:49:54.954894Z INFO text_embeddings_router: router/src/lib.rs:254: Starting model backend
2026-01-09T09:49:54.958993Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:68: Starting Python backend
2026-01-09T09:50:05.149443Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:132: Waiting for Python backend to be ready...
2026-01-09T09:50:09.033160Z INFO python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:37: backend device: npu
2026-01-09T09:50:11.528433Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:129: Python backend ready in 16.38231977s
2026-01-09T09:50:11.529131Z INFO python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:37: Server started at unix:///tmp/text-embeddings-inference-server
2026-01-09T09:50:11.683865Z ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.
Traceback (most recent call last):
File "/usr/local/bin/python-text-embeddings-server", line 7, in <module>
sys.exit(app())
File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
return get_command(self)(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 716, in main
return _main(
File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 216, in _main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
return callback(**use_params) # type: ignore
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
server.serve(model_path, dtype, uds_path, pool)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve
asyncio.run(serve_inner(model_path, dtype))
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
self._run_once()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.11/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
return await self.intercept(
> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept
return await response
File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor
raise error
File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor
return await behavior(request_or_iterator, context)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 39, in Embed
embeddings = self.model.embed(batch)
File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 91, in embed
return self._process_default(batch, kwargs)
File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 122, in _process_default
output = self.model(**kwargs)
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/Ascend/mxRag/ops/transformer_adapter/modeling_xlm_roberta_adapter.py", line 91, in forward_boost
position_ids = position_ids.view(-1, seq_length).long()
RuntimeError: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.
2026-01-09T09:50:11.684434Z ERROR health:embed:embed: backend_grpc_client: backends/grpc-client/src/lib.rs:25: Server error: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.
2026-01-09T09:50:11.732239Z INFO text_embeddings_backend_python::management: backends/python/src/management.rs:146: Python backend process terminated
Error: Model backend is not healthy
Caused by:
Server error: npu_dtype_cast_impl_op_api:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:35 NPU function error: call aclnnCast failed, error code is 561103
[ERROR] 2026-01-09-09:50:11 (PID:788, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 788] 2026-01-09-09:50:11.670.128 AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
TraceBack (most recent call last):
Failed to ParseDynamicKernels.
AclOpKernelInit failed opType
Cast ADD_TO_LAUNCHER_LIST_AICORE failed.