昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
Atlas 300I Duo卡启服务化报错：RunKernel failed! ret:3；RunAllKernel failed! ret:3；PagedAttentionOpsRunner_13_0_3:1 Execute Failed
收藏回复举报
Atlas 300I Duo卡启服务化报错：RunKernel failed! ret:3；RunAllKernel failed! ret:3；PagedAttentionOpsRunner_13_0_3:1 Execute Failed
t('forum.solved') 已解决
发表于2025-04-14 14:13:06
0 查看

MindIE版本：1.0.0

CANN：8.0.0

驱动：24.1.rc3

启服务化命令：./bin/mindieservice_daemon

加入如下参数后报错：

cke_3223.png

报错日志：

.......................

[2025-04-14 12:45:26,037] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,037] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,038] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,038] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,039] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,039] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,040] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,040] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,041] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,041] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,042] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,042] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,043] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,043] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,044] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,046] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : trans to 29

[2025-04-14 12:45:26,526] [307055] [281464795283808] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])

[2025-04-14 12:45:26,971] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 16, 128, 16])

[2025-04-14 12:45:26,974] [307039] [281463668076896] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])

[2025-04-14 12:45:26,975] [307039] [281463668076896] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281472594843632 id of vcache is 281472594843536

[2025-04-14 12:45:29,592] [307041] [281464658641248] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])

[2025-04-14 12:45:32,743] [307044] [281464187044192] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])

[2025-04-14 12:45:35.532862] [error] [307844] [paged_attention_tiling.cpp:330] prefill pa only support no mask and lookahead mask

[2025-04-14 12:45:35.532934] [error] [307844] [paged_attention_tiling.cpp:355] code:1, msg:

[2025-04-14 12:45:35.532948] [error] [307844] [kernel_base.cpp:251] Failed to init run info code:1, msg:

[2025-04-14 12:45:35.532956] [error] [307844] [mki_node_implement.cpp:180] PagedAttentionOpsRunner_3_0_3[2] InitRunInfo failed!

[2025-04-14 12:45:35.532963] [error] [307844] [ops_runner.cpp:296] PagedAttentionOpsRunner_3_0_3:0  node[2] InitRunInfo failed!

[2025-04-14 12:45:35.532979] [error] [307844] [ops_runner.cpp:272] PagedAttentionOpsRunner_3_0_3:0  node[2] fill tiling buffer fail, error code:3

[2025-04-14 12:45:35.532987] [error] [307844] [graph_runner.cpp:340] AttentionRunner_3_0:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.532994] [error] [307844] [graph_runner.cpp:340] Prefill_layerRunner_3:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.533002] [error] [307844] [operation_base.cpp:885] Prefill_layer_3 fill host tiling buffer fail

[2025-04-14 12:45:35.543358] [error] [307844] [paged_attention_tiling.cpp:330] prefill pa only support no mask and lookahead mask

[2025-04-14 12:45:35.543396] [error] [307844] [paged_attention_tiling.cpp:355] code:1, msg:

...............

[2025-04-14 12:45:35.707673] [error] [307844] [mki_node_implement.cpp:180] PagedAttentionOpsRunner_51_0_3[2] InitRunInfo failed!

[2025-04-14 12:45:35.707680] [error] [307844] [ops_runner.cpp:296] PagedAttentionOpsRunner_51_0_3:0  node[2] InitRunInfo failed!

[2025-04-14 12:45:35.707690] [error] [307844] [ops_runner.cpp:272] PagedAttentionOpsRunner_51_0_3:0  node[2] fill tiling buffer fail, error code:3

[2025-04-14 12:45:35.707698] [error] [307844] [graph_runner.cpp:340] AttentionRunner_51_0:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.707706] [error] [307844] [graph_runner.cpp:340] Prefill_layerRunner_51:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.707713] [error] [307844] [operation_base.cpp:885] Prefill_layer_51 fill host tiling buffer fail

[2025-04-14 12:45:35.709163] [error] [309165] [kernel_base.cpp:27] args size invalid

[2025-04-14 12:45:35.709204] [error] [309165] [kernel_base.cpp:298] failed to build kernel params

[2025-04-14 12:45:35.709217] [error] [309165] [mki_node_implement.cpp:211] PagedAttentionOpsRunner_3_0_3[2] PagedAttentionDecoderNzMaskKernel run fail, launchParam

:specificParam: {"batchRunStatus":[],"compressHead":false,"headDimV":0,"headSize":10,"identityM":[0],"isTriuMask":0,"kvHead":2,"kvSeqLen":[],"maskType":1,"outDataType":1,"qSeqLen":[12512,488],"quantType":0,"scaleType":0,"tor":0.0883883461356163,"type":2003}

intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[3]: dtype:int32, format:nd, dims:[2, 98], dataSize:784, data:1

intensors[4]: dtype:int32, format:nd, dims:[2], dataSize:8, data:1

intensors[5]: dtype:float16, format:fractal_nz, dims:[1, 782, 12512, 16], dataSize:313100288, data:1

intensors[6]: dtype:undefined, format:undefined, dims:[], dataSize:0, data:0

outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

st: code:-1, msg:

[2025-04-14 12:45:35.709336] [error] [309165] [ops_runner.cpp:602] RunKernel failed! ret:3

[2025-04-14 12:45:35.709345] [error] [309165] [ops_runner.cpp:397] RunAllKernel failed! ret:3

[2025-04-14 12:45:35.709355] [error] [309165] [runner.cpp:115] PagedAttentionOpsRunner_3_0_3:1 Execute Failed. st: 3

[2025-04-14 12:45:35.709368] [error] [309165] [graph_runner.cpp:906] AttentionRunner_3_0:0  node[3] execute fail, runner name:PagedAttentionOpsRunner

[2025-04-14 12:45:35.709379] [error] [309165] [runner.cpp:115] AttentionRunner_3_0:1 Execute Failed. st: 3

[2025-04-14 12:45:35.709386] [error] [309165] [graph_runner.cpp:906] Prefill_layerRunner_3:0  node[0] execute fail, runner name:AttentionRunner

[2025-04-14 12:45:35.709397] [error] [309165] [runner.cpp:115] Prefill_layerRunner_3:1 Execute Failed. st: 3

[2025-04-14 12:45:35.709405] [error] [309165] [operation_base.cpp:726] Prefill_layer_3 execute Prefill_layerRunner fail

[2025-04-14 12:45:35.709413] [error] [309165] [operation_base.cpp:764] Prefill_layer_3 execute fail, error code: 3

[2025-04-14 12:45:35.709696] [error] [307846] [paged_attention_tiling.cpp:330] prefill pa only support no mask and lookahead mask

[2025-04-14 12:45:35.709725] [error] [307846] [paged_attention_tiling.cpp:355] code:1, msg:

[2025-04-14 12:45:35.709737] [error] [307846] [kernel_base.cpp:251] Failed to init run info code:1, msg:

[2025-04-14 12:45:35.709735] [error] [309165] [kernel_base.cpp:27] args size invalid

[2025-04-14 12:45:35.709746] [error] [307846] [mki_node_implement.cpp:180] PagedAttentionOpsRunner_7_0_3[2] InitRunInfo failed!

[2025-04-14 12:45:35.709752] [error] [309165] [kernel_base.cpp:298] failed to build kernel params

[2025-04-14 12:45:35.709756] [error] [307846] [ops_runner.cpp:296] PagedAttentionOpsRunner_7_0_3:0  node[2] InitRunInfo failed!

[2025-04-14 12:45:35.709767] [error] [307846] [ops_runner.cpp:272] PagedAttentionOpsRunner_7_0_3:0  node[2] fill tiling buffer fail, error code:3

[2025-04-14 12:45:35.709778] [error] [307846] [graph_runner.cpp:340] AttentionRunner_7_0:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.709788] [error] [307846] [graph_runner.cpp:340] Prefill_layerRunner_7:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.709797] [error] [307846] [operation_base.cpp:885] Prefill_layer_7 fill host tiling buffer fail

[2025-04-14 12:45:35.709758] [error] [309165] [mki_node_implement.cpp:211] PagedAttentionOpsRunner_4_0_3[2] PagedAttentionDecoderNzMaskKernel run fail, launchParam

:specificParam: {"batchRunStatus":[],"compressHead":false,"headDimV":0,"headSize":10,"identityM":[0],"isTriuMask":0,"kvHead":2,"kvSeqLen":[],"maskType":1,"outDataType":1,"qSeqLen":[12512,488],"quantType":0,"scaleType":0,"tor":0.0883883461356163,"type":2003}

intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[3]: dtype:int32, format:nd, dims:[2, 98], dataSize:784, data:1

intensors[4]: dtype:int32, format:nd, dims:[2], dataSize:8, data:1

intensors[5]: dtype:float16, format:fractal_nz, dims:[1, 782, 12512, 16], dataSize:313100288, data:1

intensors[6]: dtype:undefined, format:undefined, dims:[], dataSize:0, data:0

outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

..............

[2025-04-14 12:45:35.714712] [error] [307844] [paged_attention_tiling.cpp:355] code:1, msg:

[2025-04-14 12:45:35.714722] [error] [307844] [kernel_base.cpp:251] Failed to init run info code:1, msg:

[2025-04-14 12:45:35.714729] [error] [307844] [mki_node_implement.cpp:180] PagedAttentionOpsRunner_53_0_3[2] InitRunInfo failed!

[2025-04-14 12:45:35.714736] [error] [307844] [ops_runner.cpp:296] PagedAttentionOpsRunner_53_0_3:0  node[2] InitRunInfo failed!

[2025-04-14 12:45:35.714746] [error] [307844] [ops_runner.cpp:272] PagedAttentionOpsRunner_53_0_3:0  node[2] fill tiling buffer fail, error code:3

[2025-04-14 12:45:35.714754] [error] [307844] [graph_runner.cpp:340] AttentionRunner_53_0:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.714761] [error] [307844] [graph_runner.cpp:340] Prefill_layerRunner_53:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.714769] [error] [307844] [operation_base.cpp:885] Prefill_layer_53 fill host tiling buffer fail

[2025-04-14 12:45:35.715520] [error] [307846] [paged_attention_tiling.cpp:330] prefill pa only support no mask and lookahead mask

[2025-04-14 12:45:35.715554] [error] [307846] [paged_attention_tiling.cpp:355] code:1, msg:

[2025-04-14 12:45:35.715565] [error] [307846] [kernel_base.cpp:251] Failed to init run info code:1, msg:

[2025-04-14 12:45:35.715574] [error] [307846] [mki_node_implement.cpp:180] PagedAttentionOpsRunner_9_0_3[2] InitRunInfo failed!

[2025-04-14 12:45:35.715581] [error] [307846] [ops_runner.cpp:296] PagedAttentionOpsRunner_9_0_3:0  node[2] InitRunInfo failed!

[2025-04-14 12:45:35.715592] [error] [307846] [ops_runner.cpp:272] PagedAttentionOpsRunner_9_0_3:0  node[2] fill tiling buffer fail, error code:3

[2025-04-14 12:45:35.715601] [error] [307846] [graph_runner.cpp:340] AttentionRunner_9_0:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

2025-04-14 12:45:35,710 [ERROR] model.py:39 - [Model]   >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize

    return self.python_model.initialize(config)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize

    self.generator = Generator(

                     ^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__

    self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up

    raise e

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up

    self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend

    self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 181, in warm_up

    super().warm_up(model_inputs, q_lens=q_lens)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up

    _ = self.forward(model_inputs, **kwargs)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward

    logits = self.forward_tensor(

             ^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor

    logits = self.model_runner.forward(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

    return self.model.forward(**kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

    acl_model_out = model_operation.execute(acl_inputs, acl_param)

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

[2025-04-14 12:45:35.715615] [error] [307846] [graph_runner.cpp:340] Prefill_layerRunner_9:0 GraphRunner::FillHostTilingBufferImpl failed! error code:3

[2025-04-14 12:45:35.715677] [error] [307846] [operation_base.cpp:885] Prefill_layer_9 fill host tiling buffer fail

2025-04-14 12:45:35,715 [ERROR] model.py:42 - [Model]   >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}

[2025-04-14 12:45:35.716156] [error] [308860] [kernel_base.cpp:27] args size invalid

[2025-04-14 12:45:35.716194] [error] [308860] [kernel_base.cpp:298] failed to build kernel params

[2025-04-14 12:45:35.716203] [error] [308860] [mki_node_implement.cpp:211] PagedAttentionOpsRunner_3_0_3[2] PagedAttentionDecoderNzMaskKernel run fail, launchParam

:specificParam: {"batchRunStatus":[],"compressHead":false,"headDimV":0,"headSize":10,"identityM":[0],"isTriuMask":0,"kvHead":2,"kvSeqLen":[],"maskType":1,"outDataType":1,"qSeqLen":[12512,488],"quantType":0,"scaleType":0,"tor":0.0883883461356163,"type":2003}

intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 16, 128, 16], dataSize:65536, data:1

intensors[3]: dtype:int32, format:nd, dims:[2, 98], dataSize:784, data:1

intensors[4]: dtype:int32, format:nd, dims:[2], dataSize:8, data:1

intensors[5]: dtype:float16, format:fractal_nz, dims:[1, 782, 12512, 16], dataSize:313100288, data:1

intensors[6]: dtype:undefined, format:undefined, dims:[], dataSize:0, data:0

outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 80, 13008, 16], dataSize:33300480, data:1

..............

st: code:-1, msg:

[2025-04-14 12:45:35.728138] [error] [308996] [ops_runner.cpp:602] RunKernel failed! ret:3

[2025-04-14 12:45:35.728144] [error] [308996] [ops_runner.cpp:397] RunAllKernel failed! ret:3

[2025-04-14 12:45:35.728150] [error] [308996] [runner.cpp:115] PagedAttentionOpsRunner_13_0_3:1 Execute Failed. st: 3

[2025-04-14 12:45:35.728158] [error] [308996] [graph_runner.cpp:906] AttentionRunner_13_0:0  node[3] execute fail, runner name:PagedAttentionOpsRunner

[2025-04-14 12:45:35.728165] [error] [308996] [runner.cpp:115] AttentionRunner_13_0:1 Execute Failed. st: 3

[2025-04-14 12:45:35.728172] [error] [308996] [graph_runner.cpp:906] Prefill_layerRunner_13:0  node[0] execute fail, runner name:AttentionRunner

[2025-04-14 12:45:35.728179] [error] [308996] [runner.cpp:115] Prefill_layerRunner_13:1 Execute Failed. st: 3

[2025-04-14 12:45:35.728185] [error] [308996] [operation_base.cpp:726] Prefill_layer_13 execute Prefill_layerRunner fail

[2025-04-14 12:45:35.728191] [error] [308996] [operation_base.cpp:764] Prefill_layer_13 execute fail, error code: 3

2025-04-14 12:45:35,726 [ERROR] model.py:39 - [Model]   >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize

    return self.python_model.initialize(config)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize

    self.generator = Generator(

                     ^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__

    self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up

    raise e

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up

    self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend

    self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 181, in warm_up

    super().warm_up(model_inputs, q_lens=q_lens)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up

    _ = self.forward(model_inputs, **kwargs)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward

    logits = self.forward_tensor(

             ^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor

    logits = self.model_runner.forward(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

    return self.model.forward(**kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

    acl_model_out = model_operation.execute(acl_inputs, acl_param)

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

2025-04-14 12:45:35,730 [ERROR] model.py:42 - [Model]   >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}

2025-04-14 12:45:35,728 [ERROR] model.py:39 - [Model]   >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize

    return self.python_model.initialize(config)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize

    self.generator = Generator(

                     ^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__

    self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up

    raise e

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up

    self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend

    self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 181, in warm_up

    super().warm_up(model_inputs, q_lens=q_lens)

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up

    _ = self.forward(model_inputs, **kwargs)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward

    logits = self.forward_tensor(

             ^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor

    logits = self.model_runner.forward(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

    return self.model.forward(**kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

    acl_model_out = model_operation.execute(acl_inputs, acl_param)

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

2025-04-14 12:45:35,732 [ERROR] model.py:42 - [Model]   >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown

  warnings.warn('resource_tracker: There appear to be %d '

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown

  warnings.warn('resource_tracker: There appear to be %d '

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown

  warnings.warn('resource_tracker: There appear to be %d '

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown

  warnings.warn('resource_tracker: There appear to be %d '

Daemon is killing...

Killed

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情