尝试使用torch-npu在npu上推理VideoLLaMA3模型,推理代码中使用了FlashAttention2,运行推理代码会提示未安装FlashAttention2。搜索到的信息NPU不支持安装FlashAttention2,能否有方法修改或者替换代码以实现推理:
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
device_map={"": "cuda:1"},
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
报错信息:
Traceback (most recent call last):
File "/root/deepseek/VideoLLaMA3/inference/example_videollama3.py", line 11, in <module>
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4091, in from_pretrained
config = cls._autoset_attn_implementation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1617, in _autoset_attn_implementation
cls._check_and_enable_flash_attn_2(
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1747, in _check_and_enable_flash_attn_2
raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}")
ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2.
[ERROR] 2025-05-28-13:25:12 (PID:60427, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
/usr/lib64/python3.11/tempfile.py:904: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpabbiqtnx'>
_warnings.warn(warn_message, ResourceWarning)
npu信息:

尝试使用torch-npu在npu上推理VideoLLaMA3模型,推理代码中使用了FlashAttention2,运行推理代码会提示未安装FlashAttention2。搜索到的信息NPU不支持安装FlashAttention2,能否有方法修改或者替换代码以实现推理:
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
device_map={"": "cuda:1"},
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
报错信息:
Traceback (most recent call last):
File "/root/deepseek/VideoLLaMA3/inference/example_videollama3.py", line 11, in <module>
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4091, in from_pretrained
config = cls._autoset_attn_implementation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1617, in _autoset_attn_implementation
cls._check_and_enable_flash_attn_2(
File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1747, in _check_and_enable_flash_attn_2
raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}")
ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2.
[ERROR] 2025-05-28-13:25:12 (PID:60427, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
/usr/lib64/python3.11/tempfile.py:904: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpabbiqtnx'>
_warnings.warn(warn_message, ResourceWarning)
npu信息: