[2025-08-21 17:01:32.927048] [debug] [546] [kernel_base.cpp:324] kernel param: 8 in/out, 0 const in, 0 workspaces, 0 hwsync
[2025-08-21 17:01:32.927056] [error] [546] [kernel_base.cpp:37] kernel info args is nullptr, please check first error before.
[2025-08-21 17:01:32.927062] [error] [546] [kernel_base.cpp:332] build args failed, abort running
[2025-08-21 17:01:32.927067] [error] [546] [mki_node_implement.cpp:232] SelfAttentionEncoderFusionOpsRunner910A_33_4[3] UnpadFlashAttentionNzEncoderKernel run fail, launchParam
:specificParam: {"alibiLeftAlign":0,"batchRunStatus":[],"cacheType":0,"compressHead":false,"dataDimOrder":0,"headSize":8,"isAlibiMaskSqrt":0,"isTriuMask":0,"kvHead":8,"kvSeqLen":[64,64,64,......],"scaleType":0,"tor":0.11180339753627777,"type":2005,"windowSize":0}
intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[3]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[4]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[5]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[6]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1


st: code:1, msg:
[2025-08-21 17:01:32.927416] [error] [546] [ops_runner.cpp:655] RunKernel failed! ret:1
[2025-08-21 17:01:32.927423] [error] [546] [ops_runner.cpp:415] RunAllKernel failed! ret:1
[2025-08-21 17:01:32.927428] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927433] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927438] [error] [546] [runner.cpp:132] SelfAttentionEncoderFusionOpsRunner910A_33_4:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927446] [error] [546] [graph_runner.cpp:976] Qwen25VL_VIT_layer_0_graphRunner_33:3  node[4] execute fail, runner name:SelfAttentionEncoderFusionOpsRunner910A
[2025-08-21 17:01:32.927454] [info] [546] [graph_runner.cpp:396] Qwen25VL_VIT_layer_0_graphRunner_33:3 execute fail
[2025-08-21 17:01:32.927460] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927464] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927467] [error] [546] [runner.cpp:132] Qwen25VL_VIT_layer_0_graphRunner_33:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927473] [error] [546] [operation_base.cpp:1023] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner fail
[2025-08-21 17:01:32.927479] [info] [546] [operation_base.cpp:1033] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner success
[2025-08-21 17:01:32.927485] [info] [546] [operation_base.cpp:1041] Qwen25VL_VIT_layer_0_graph_33 execute statistic:totalTime:5428, streamSyncTime:0, tillingCopyTime:0, kernelExecuteTime:3325, launchTime:3763, preLaunchTime:1665
[2025-08-21 17:01:32.927495] [error] [546] [operation_base.cpp:1100] Qwen25VL_VIT_layer_0_graph_33 Launch fail, error code: 1
[2025-08-21 17:01:32.927] [264] [281456090214752] [llmmodels] [ERROR] [graph_operation.cpp:445] [MIE05E000002] Qwen25VL_VIT_graph nodes[1] atb operation execute fail, error:1
[2025-08-21 17:01:32,927] [264] [281456090214752] [llm] [ERROR] [logging.py-346] : Unknown exception: Atb operation setup fail
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 115
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 116
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 117
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 118
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 119
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 120
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 121
[2025-08-21 17:01:32.021+0800] [264] [281463433982304] [batchscheduler] [ERROR] [standard_model.py:197] : [Model]       >>> global rank-0 Execute type:1, Exception:Atb operation setup fail
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 185, in execute
    return self._prefill(requests)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 430, in _prefill
    return self.__generate(requests, is_prefill=True, is_mix=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 578, in __generate
    generate_output = self.__handle_requests(requests, is_prefill, is_mix)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 939, in __handle_requests
    generate_output = self.generator.generate_token(metadata)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 386, in generate_token
    raise e
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 367, in generate_token
    generation_output = self.plugin.generate_token(input_metadata)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 91, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin_manager.py", line 115, in generate_token
    result = self.generator_backend.forward(model_inputs, q_lens=self.plugin_data_param.q_len,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward
    logits = self._forward(model_inputs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward
    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward
    result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs
    result = self.forward_tensor(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor
    result = self.model_runner.forward(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward
    res = self.model.forward(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 118, in forward
    inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts = self.prepare_prefill_token_service(
                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 197, in prepare_prefill_token_service
    image_features = self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 872, in forward
    vision_features = self.encoder(
                      ^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 719, in forward
    hidden_states = self.graph.forward(self.graph_inputs, self.graph_outputs, self.graph_param)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Atb operation setup fail