背景是这样的：

我在910B机器上利用mis-tei部署embedding和rerank服务，运行一段时间后，就 开始疯狂报下面的错误，更换模型或者重启服务后也是如此。请大佬帮帮忙吧 T T...

报错内容如下：

ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.

Traceback (most recent call last):

  File "/usr/local/bin/python-text-embeddings-server", line 8, in <module>

    sys.exit(app())

  File "/usr/local/lib/python3.11/dist-packages/typer/main.py", line 311, in __call__

    return get_command(self)(*args, **kwargs)

  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.11/dist-packages/typer/core.py", line 716, in main

    return _main(

  File "/usr/local/lib/python3.11/dist-packages/typer/core.py", line 216, in _main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke

    return __callback(*args, **kwargs)

  File "/usr/local/lib/python3.11/dist-packages/typer/main.py", line 683, in wrapper

    return callback(**use_params)  # type: ignore

  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve

    server.serve(model_path, dtype, uds_path, pool)

  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve

    asyncio.run(serve_inner(model_path, dtype))

  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run

    return runner.run(main)

  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run

    return self._loop.run_until_complete(task)

  File "/usr/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete

    self.run_forever()

  File "/usr/lib/python3.11/asyncio/base_events.py", line 608, in run_forever

    self._run_once()

  File "/usr/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once

    handle._run()

  File "/usr/lib/python3.11/asyncio/events.py", line 84, in _run

    self._context.run(self._callback, *self._args)

  File "/usr/local/lib/python3.11/dist-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method

    return await self.intercept(

> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept

    return await response

  File "/usr/local/lib/python3.11/dist-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor

    raise error

  File "/usr/local/lib/python3.11/dist-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor

    return await behavior(request_or_iterator, context)

  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 35, in Embed

    batch = self.model.batch_type.from_pb(

  File "/usr/lib/python3.11/contextlib.py", line 81, in inner

    return func(*args, **kwds)

  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/types.py", line 75, in from_pb

    all_tensors = all_tensors.to(device)

RuntimeError: ACL stream synchronize failed, error code:507015

ERROR backend_task:embed:embed: backend_grpc_client: backends/grpc-client/src/lib.rs:25: Server error: ACL stream synchronize failed, error code:507015

ERROR embed:embed_pooled{truncate=true truncation_direction=Right prompt_name=None normalize=true}: text_embeddings_core::infer: core/src/infer.rs:361: Server error: ACL stream synchronize failed, error code:507015