大模型使用FlashAttention2怎么在NPU上实现推理
收藏回复举报
大模型使用FlashAttention2怎么在NPU上实现推理
t('forum.solved') 已解决
新人帖
发表于2025-05-28 14:04:56
0 查看

尝试使用torch-npu在npu上推理VideoLLaMA3模型,推理代码中使用了FlashAttention2,运行推理代码会提示未安装FlashAttention2。搜索到的信息NPU不支持安装FlashAttention2,能否有方法修改或者替换代码以实现推理:

model = AutoModelForCausalLM.from_pretrained( 

    model_path, 

    trust_remote_code=True, 

    device_map={"": "cuda:1"}, 

    torch_dtype=torch.bfloat16, 

    attn_implementation="flash_attention_2", 

)

报错信息:

Traceback (most recent call last): 

  File "/root/deepseek/VideoLLaMA3/inference/example_videollama3.py", line 11, in <module> 

    model = AutoModelForCausalLM.from_pretrained( 

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained 

    return model_class.from_pretrained( 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4091, in from_pretrained 

    config = cls._autoset_attn_implementation( 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1617, in _autoset_attn_implementation 

    cls._check_and_enable_flash_attn_2( 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1747, in _check_and_enable_flash_attn_2 

    raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}") 

ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. 

[ERROR] 2025-05-28-13:25:12 (PID:60427, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception 

/usr/lib64/python3.11/tempfile.py:904: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpabbiqtnx'> 

  _warnings.warn(warn_message, ResourceWarning) 

npu信息:

cke_47255.png

我要发帖子