运行Huggingface权重转换到Megatron-LM格式的脚本bash examples/llama2/ckpt_convert_llama2_hf2legacy.sh时出现了如下报错:
在gitee/torchair的issues下看到也有同样问题,下面给出的解决方法是:将CANN升级到8.0版本(我所使用的910A上预设CANN版本是7.0.RC1的)
(fbig) [ma-user ModelLink]$bash examples/llama2/ckpt_convert_llama2_hf2legacy.sh
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/utils/path_manager.py:79: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current user.
warnings.warn(f"Warning: The {path} owner does not match the current user.")
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/utils/path_manager.py:79: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/7.0.0.alpha001/aarch64-linux/ascend_toolkit_install.info owner does not match the current user.
warnings.warn(f"Warning: The {path} owner does not match the current user.")
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:211: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:171: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
warnings.warn(msg, RuntimeWarning)
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:171: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
warnings.warn(msg, RuntimeWarning)
Traceback (most recent call last):
File "convert_ckpt.py", line 23, in <module>
import modellink
File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/__init__.py", line 24, in <module>
from . import patchs
File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/__init__.py", line 21, in <module>
exec_adaptation()
File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 69, in exec_adaptation
patch_megatron_core()
File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 93, in patch_megatron_core
patch_core_models(modellink_args)
File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 129, in patch_core_models
from mindspeed.core.models.common.embeddings.rotary_pos_embedding import get_pos_emb_on_this_cp_rank
File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/core/models/common/embeddings/rotary_pos_embedding.py", line 12, in <module>
from mindspeed.ops.npu_rotary_position_embedding import npu_rotary_position_embedding
File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/ops/npu_rotary_position_embedding.py", line 2, in <module>
from mindspeed.op_builder import RotaryPositionEmbeddingOpBuilder
File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/__init__.py", line 11, in <module>
from .gmm_builder import GMMOpBuilder, GMMV2OpBuilder
File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/gmm_builder.py", line 18, in <module>
from torchair.ge_concrete_graph.fx2ge_converter import register_fx_node_ge_converter
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 17, in __getattr__
raise self._e
File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/gmm_builder.py", line 12, in <module>
from torchair import ge
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 17, in __getattr__
raise self._e
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 36, in _get_default_backend
from . import torchair
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/__init__.py", line 5, in <module>
from torchair.npu_fx_compiler import get_compiler
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/__init__.py", line 5, in <module>
from torchair.npu_fx_compiler import get_compiler
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/npu_fx_compiler.py", line 31, in <module>
from torchair.ge_concrete_graph.fx2ge_converter import GeConcreteGraph as ConcreteGraph
File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/ge_concrete_graph/fx2ge_converter.py", line 22, in <module>
from torchair.core import _torchair
ImportError: /home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/core/_abi_compat_ge_apis.so: undefined symbol: _ZN2ge5Graph28LoadFromSerializedModelArrayEPKvm
/home/ma-user/anaconda3/envs/fbig/lib/python3.8/tempfile.py:827: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2aigp1a7'>
_warnings.warn(warn_message, ResourceWarning)
运行环境:torch=2.1.0,torch_npu=2.1.0.post3,cann=8.0.RC1
问题描述:尝试在910上完成llama2-7B的预训练和推理部署,参考modelLink使用指南文档:examples/README.md · Ascend/ModelLink - Gitee.com选择了llama2-7B大模型熟悉适配流程
运行Huggingface权重转换到Megatron-LM格式的脚本bash examples/llama2/ckpt_convert_llama2_hf2legacy.sh时出现了如下报错:
在gitee/torchair的issues下看到也有同样问题,下面给出的解决方法是:将CANN升级到8.0版本(我所使用的910A上预设CANN版本是7.0.RC1的)
可是我升级到CANN8.0.RC1并重新安装开发环境后,仍旧存在该报错。详细信息如下:
请问该如何有效解决该报错。