昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
(EZ9999): The error from device(2), serial number is 7, there is an error of aivec, core id is 3, error code = 0x800000
收藏回复举报
(EZ9999): The error from device(2), serial number is 7, there is an error of aivec, core id is 3, error code = 0x800000
t('forum.solved') 已解决
发表于2026-07-02 08:55:25
0 查看

使用mindie2.3.0的docker镜像，运行Qwen3-30B-A3B的模型

显卡信息如下

cke_1032.png    ​

docker-compose.yml文件如下

services:
  deepseek-r1:
    container_name: Qwen3-30B-A3B
    shm_size: "8g"
    image: 02efebd6ce64
    user: root
    privileged: true
    network_mode: host
    restart: unless-stopped
    stdin_open: true
    tty: true
    command: >
      /bin/bash -c 
      "
      nohup ./bin/mindieservice_daemon > logs/Qwen3-30B-A3B-output.log 2>&1 &
      tail -f logs/Qwen3-30B-A3B-output.log
      "
    working_dir: "/usr/local/Ascend/mindie/latest/mindie-service"
    environment: 
      MINDIE_LOG_TO_STDOUT: "benchmark:1; client:1"
      LD_LIBRARY_PATH: "/usr/local/lib64/python3.11/site-packages/torch/lib:/usr/lib64:/lib:/lib:/usr/local/Ascend/atb-models/lib:/usr/local/Ascend/mindie/latest/mindie-llm/lib:/usr/local/Ascend/mindie/latest/mindie-llm/lib/grpc:/usr/local/Ascend/mindie/latest/mindie-service/lib:/usr/local/Ascend/mindie/latest/mindie-service/lib/grpc:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64/common:"
    devices:
      - "/dev/davinci_manager:/dev/davinci_manager"
      - "/dev/devmm_svm:/dev/devmm_svm"
      - "/dev/hisi_hdc:/dev/hisi_hdc"
    volumes:
      - "/usr/local/Ascend/driver:/usr/local/Ascend/driver"
      - "/usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons"
      - "/usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi"
      - "/usr/local/sbin/:/usr/local/sbin/"
      - "/var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf"
      - "/var/log/npu/slog/:/var/log/npu/slog"
      - "/var/log/npu/profiling/:/var/log/npu/profiling"
      - "/var/log/npu/dump/:/var/log/npu/dump"
      - "/var/log/npu/:/usr/slog"
      - "/data/llm_models:/data/llm_models"
      - "/home/HwHiAiUser/mindie/Qwen3-30B-A3B/config.json:/usr/local/Ascend/mindie/latest/mindie-service/conf/config.json"
      - "/home/HwHiAiUser/mindie/Qwen3-30B-A3B/logs:/usr/local/Ascend/mindie/latest/mindie-service/logs"

mindie中config模型启动配置如下

{
    "Version" : "1.0.0",

    "ServerConfig" :
    {
        "ipAddress" : "0.0.0.0",
        "managementIpAddress" : "0.0.0.0",
        "port" : 1045,
        "managementPort" : 1046,
        "metricsPort" : 1047,
        "allowAllZeroIpListening" : true,
        "maxLinkNum" : 1000,
        "httpsEnabled" : false,
        "fullTextEnabled" : false,
        "tlsCaPath" : "security/ca/",
        "tlsCaFile" : ["ca.pem"],
        "tlsCert" : "security/certs/server.pem",
        "tlsPk" : "security/keys/server.key.pem",
        "tlsPkPwd" : "security/pass/key_pwd.txt",
        "tlsCrlPath" : "security/certs/",
        "tlsCrlFiles" : ["server_crl.pem"],
        "managementTlsCaFile" : ["management_ca.pem"],
        "managementTlsCert" : "security/certs/management/server.pem",
        "managementTlsPk" : "security/keys/management/server.key.pem",
        "managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
        "managementTlsCrlPath" : "security/management/certs/",
        "managementTlsCrlFiles" : ["server_crl.pem"],
        "metricsTlsCaFile" : ["metrics_ca.pem"],
        "metricsTlsCert" : "security/certs/metrics/server.pem",
        "metricsTlsPk" : "security/keys/metrics/server.key.pem",
        "metricsTlsPkPwd" : "security/pass/metrics/key_pwd.txt",
        "metricsTlsCrlPath" : "security/metrics/certs/",
        "metricsTlsCrlFiles" : ["server_crl.pem"],
        "kmcKsfMaster" : "tools/pmt/master/ksfa",
        "kmcKsfStandby" : "tools/pmt/standby/ksfb",
        "inferMode" : "standard",
        "interCommTLSEnabled" : true,
        "interCommPort" : 1121,
        "interCommTlsCaPath" : "security/grpc/ca/",
        "interCommTlsCaFiles" : ["ca.pem"],
        "interCommTlsCert" : "security/grpc/certs/server.pem",
        "interCommPk" : "security/grpc/keys/server.key.pem",
        "interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
        "interCommTlsCrlPath" : "security/grpc/certs/",
        "interCommTlsCrlFiles" : ["server_crl.pem"],
        "openAiSupport" : "vllm",
        "tokenTimeout" : 600,
        "e2eTimeout" : 600,
        "distDPServerEnabled":false
    },

    "BackendConfig" : {
        "backendName" : "mindieservice_llm_engine",
        "modelInstanceNumber" : 1,
        "npuDeviceIds" : [[0,1,2,3]],
        "tokenizerProcessNumber" : 8,
        "multiNodesInferEnabled" : false,
        "multiNodesInferPort" : 1120,
        "interNodeTLSEnabled" : true,
        "interNodeTlsCaPath" : "security/grpc/ca/",
        "interNodeTlsCaFiles" : ["ca.pem"],
        "interNodeTlsCert" : "security/grpc/certs/server.pem",
        "interNodeTlsPk" : "security/grpc/keys/server.key.pem",
        "interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
        "interNodeTlsCrlPath" : "security/grpc/certs/",
        "interNodeTlsCrlFiles" : ["server_crl.pem"],
        "interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
        "interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
        "kvPoolConfig" : {"backend":"", "configPath":""},
        "ModelDeployConfig" :
        {
            "maxSeqLen" : 20480,
            "maxInputTokenLen" : 10240,
            "truncation" : false,
            "ModelConfig" : [
                {
                    "modelInstanceType" : "Standard",
                    "modelName" : "Qwen3-30B-A3B",
                    "modelWeightPath" : "/data/llm_models/Qwen/Qwen3-30B-A3B",
                    "worldSize" : 4,
                    "cpuMemSize" : 0,
                    "npuMemSize" : -1,
                    "backendType" : "atb",
                    "trustRemoteCode" : false,
                    "async_scheduler_wait_time": 120,
                    "kv_trans_timeout": 10,
                    "kv_link_timeout": 1080
                }
            ]
        },

        "ScheduleConfig" :
        {
            "templateType" : "Standard",
            "templateName" : "Standard_LLM",
            "cacheBlockSize" : 128,

            "maxPrefillBatchSize" : 50,
            "maxPrefillTokens" : 10240,
            "prefillTimeMsPerReq" : 150,
            "prefillPolicyType" : 0,

            "decodeTimeMsPerReq" : 50,
            "decodePolicyType" : 0,

            "maxBatchSize" : 200,
            "maxIterTimes" : 10240,
            "maxPreemptCount" : 0,
            "supportSelectBatch" : false,
            "maxQueueDelayMicroseconds" : 5000,
            "maxFirstTokenWaitTime": 2500
        }
    },

    "LogConfig": {
        "dynamicLogLevel" : "",
        "dynamicLogLevelValidHours" : 2,
        "dynamicLogLevelValidTime" : ""
    },

    "EnableDynamicAdjustTimeoutConfig": false
}

docker日志报错如下

g_mainPid = 7
LogLevelDynamicHandler start
ConfigManager: Load Config from /usr/local/Ascend/mindie/2.3.0/mindie-service/conf/config.json.
The configName speculationGamma is not found, use default value.
[ConfigManager::ExecuteConfigInteractions] Configuration interactions completed successfully
[ConfigManager::InitConfigManager] Successfully init config manager
LogLevelDynamicHandler start
LogLevelDynamicHandler start
LogLevelDynamicHandler start
LogLevelDynamicHandler start

Loading selected layers:   0%|          | 0/48 [00:00<?, ?layer/s]
Loading selected layers:   2%|▏         | 1/48 [00:01<00:47,  1.01s/layer]
Loading selected layers:   4%|▍         | 2/48 [00:01<00:44,  1.02layer/s]
Loading selected layers:   6%|▋         | 3/48 [00:02<00:44,  1.02layer/s]
Loading selected layers:   8%|▊         | 4/48 [00:03<00:43,  1.01layer/s]
Loading selected layers:  10%|█         | 5/48 [00:04<00:42,  1.00layer/s]
Loading selected layers:  12%|█▎        | 6/48 [00:05<00:41,  1.00layer/s]
Loading selected layers:  15%|█▍        | 7/48 [00:06<00:41,  1.00s/layer]
Loading selected layers:  17%|█▋        | 8/48 [00:07<00:40,  1.01s/layer]
Loading selected layers:  19%|█▉        | 9/48 [00:09<00:39,  1.01s/layer]
Loading selected layers:  21%|██        | 10/48 [00:10<00:38,  1.01s/layer]
Loading selected layers:  23%|██▎       | 11/48 [00:11<00:37,  1.00s/layer]
Loading selected layers:  25%|██▌       | 12/48 [00:12<00:36,  1.00s/layer]
Loading selected layers:  27%|██▋       | 13/48 [00:13<00:37,  1.06s/layer]
Loading selected layers:  29%|██▉       | 14/48 [00:14<00:38,  1.14s/layer]
Loading selected layers:  31%|███▏      | 15/48 [00:15<00:39,  1.19s/layer]
Loading selected layers:  33%|███▎      | 16/48 [00:17<00:39,  1.23s/layer]
Loading selected layers:  35%|███▌      | 17/48 [00:18<00:39,  1.26s/layer]
Loading selected layers:  38%|███▊      | 18/48 [00:19<00:38,  1.29s/layer]
Loading selected layers:  40%|███▉      | 19/48 [00:21<00:37,  1.28s/layer]
Loading selected layers:  42%|████▏     | 20/48 [00:22<00:33,  1.20s/layer]
Loading selected layers:  44%|████▍     | 21/48 [00:23<00:31,  1.15s/layer]
Loading selected layers:  46%|████▌     | 22/48 [00:24<00:28,  1.11s/layer]
Loading selected layers:  48%|████▊     | 23/48 [00:25<00:26,  1.08s/layer]
Loading selected layers:  50%|█████     | 24/48 [00:26<00:25,  1.06s/layer]
Loading selected layers:  52%|█████▏    | 25/48 [00:27<00:23,  1.04s/layer]
Loading selected layers:  54%|█████▍    | 26/48 [00:28<00:22,  1.02s/layer]
Loading selected layers:  56%|█████▋    | 27/48 [00:29<00:21,  1.02s/layer]
Loading selected layers:  58%|█████▊    | 28/48 [00:30<00:20,  1.01s/layer]
Loading selected layers:  60%|██████    | 29/48 [00:31<00:20,  1.06s/layer]
Loading selected layers:  62%|██████▎   | 30/48 [00:32<00:20,  1.14s/layer]
Loading selected layers:  65%|██████▍   | 31/48 [00:33<00:20,  1.19s/layer]
Loading selected layers:  67%|██████▋   | 32/48 [00:35<00:19,  1.19s/layer]
Loading selected layers:  69%|██████▉   | 33/48 [00:36<00:16,  1.13s/layer]
Loading selected layers:  71%|███████   | 34/48 [00:37<00:15,  1.09s/layer]
Loading selected layers:  73%|███████▎  | 35/48 [00:38<00:13,  1.07s/layer]
Loading selected layers:  75%|███████▌  | 36/48 [00:39<00:12,  1.05s/layer]
Loading selected layers:  77%|███████▋  | 37/48 [00:40<00:11,  1.04s/layer]
Loading selected layers:  79%|███████▉  | 38/48 [00:41<00:10,  1.03s/layer]
Loading selected layers:  81%|LogLevelDynamicHandler start
████████▏ | 39/48 [00:42<00:09,  1.01s/layer]
Loading selected layers:  83%|████████▎ | 40/48 [00:43<00:08,  1.01s/layer]
Loading selected layers:  85%|████████▌ | 41/48 [00:44<00:06,  1.01layer/s]
Loading selected layers:  88%|████████▊ | 42/48 [00:45<00:06,  1.04s/layer]
Loading selected layers:  90%|████████▉ | 43/48 [00:46<00:05,  1.12s/layer]
Loading selected layers:  92%|█████████▏| 44/48 [0LogLevelDynamicHandler start
LogLevelDynamicHandler start
0:47<00:04,  1.17s/layer]
Loading selected layers:  94%|█████████▍| 45/48 [00:49<00:03,  1.20s/layer]
Loading selected layers:  96%|█████████▌| 46/48 [00:50<00:02,  1.23s/layer]
Loading selected layers:  98%|█████████▊| 47/48 [00:51<00:01,  1.24s/layer]
Loading selected layers: 100%|██████████| 48/48 [00:52<00:00,  1.23s/layer]
Loading selected layers: 100%|██████████| 48/48 [00:52<00:00,  1.10s/layer]
LogLevelDynamicHandler start
[W compiler_depend.ts:57] Warning: EZ9999: Inner Error!
EZ9999[PID: 93] 2026-07-02-08:48:03.301.426 (EZ9999):  The error from device(2), serial number is 8, there is an error of aivec, core id is 3, error code = 0x800000, dump info: pc start: 0x8001244002019fc, current: 0x124400201d10, vec error info: 0x1bef717d, mte error info: 0x30000b6, ifu error info: 0xfd5d785edd80, ccu error info: 0x7f4fdfef003ca06b, cube error info: 0, biu error info: 0, aic error mask: 0x6de01200c0122cc, para base: 0x12c0c0800800, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:46]
        TraceBack (most recent call last):
       The device(2), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_core_proc.cc][LINE:105]
       coreId( 0):        0x800000    [FUNC:PrintCoreInfoErrMsg][FILE:device_error_core_proc.cc][LINE:119]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1555]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1191]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1191]
       Aicore kernel execute failed, device_id=2, stream_id=11, report_stream_id=11, task_id=2, flip_num=0, fault kernel_name=RealDiv_5c5e58e043224cccf51a06108010ae13_high_precision_223000000_mix_aic, fault kernel info ext=none, program id=3, hash=1866892713628807152.[FUNC:GetError][FILE:stream.cc][LINE:1191]
       rtStreamSynchronize execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]


DEVICE[2] PID[93]: 
EXCEPTION STREAM:
  Exception info:TGID=517259, model id=65535, stream id=11, stream phase=SCHEDULE
  Message info[0]:RTS_HWTS: Vector core exception, slot_id=17, stream_id=11
    Other info[0]:time=2026-07-02-08:47:47.659.869, function=process_hwts_error_exception, line=646, error code=0x31 (function copy_between_host_and_device_opapi)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/status.py", line 17, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 85, in do_inference
    self.initialize(execute_request.config)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 70, in initialize
    initialize_result = self.router_impl.initialize(config)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 168, in initialize
    self.generator = Generator(model_config={**model_config.model_config})
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 291, in __init__
    self.cache_manager = self.warm_up(
                         ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 485, in warm_up
    npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 791, in __warmup_standard
    npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 779, in __warmup_prefill
    npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 880, in __auto_warmup
    self.__execute_warm_up(cache_manager, input_metadata, dummy=True)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 666, in __execute_warm_up
    raise e
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 657, in __execute_warm_up
    self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode,
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 627, in _warm_up
    super()._warm_up(model_inputs, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 270, in _warm_up
    logits = self.forward(model_inputs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 68, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 212, in forward
    logits = self._forward(model_inputs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 750, in _forward
    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 133, in forward
    result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 223, in forward_from_model_inputs
    result = self.forward_tensor(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 298, in forward_tensor
    raise e
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 283, in forward_tensor
    result = self.model_runner.forward(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 344, in forward
    res = self.model.forward(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_moe/flash_causal_qwen2_moe.py", line 438, in forward
    logits = super().forward(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 526, in forward
    acl_inputs, acl_param = self.prepare_inputs_for_ascend(input_ids, position_ids, is_prefill, kv_cache,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_moe/flash_causal_qwen2_moe.py", line 392, in prepare_inputs_for_ascend
    self.get_nz_attn_mask(kv_cache[0][0].dtype, kv_cache[0][0].device)
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_moe/flash_causal_qwen2_moe.py", line 319, in get_nz_attn_mask
    attention_mask_temp = self.attn_mask.get_attn_mask(pad_maxs, dtype, device)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/utils/layers/attention/attention_mask.py", line 65, in get_attn_mask
    self.update_attn_cache(dtype, device, max_s, mini_type)
  File "/usr/local/Ascend/atb-models/atb_llm/utils/layers/attention/attention_mask.py", line 60, in update_attn_cache
    self.atten_mask_cache = self.atten_mask_cache.to(device)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch_npu/utils/_to_int.py", line 34, in decorated
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
RuntimeError: ACL stream synchronize failed, error code:507035
Daemon is killing, please wait about 15 seconds...
Waiting for subprocess exits for 1/10 seconds.
Waiting for subprocess exits for 2/10 seconds.
Waiting for subprocess exits for 3/10 seconds.
Waiting for subprocess exits for 4/10 seconds.
Waiting for subprocess exits for 5/10 seconds.
Waiting for subprocess exits for 6/10 seconds.
Waiting for subprocess exits for 7/10 seconds.
Waiting for subprocess exits for 8/10 seconds.
Waiting for subprocess exits for 9/10 seconds.
Waiting for subprocess exits for 10/10 seconds.
Timeout reached, some subprocesses still alive. Force to kill them.

 

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情