910A上运行Modellink中llama2-7B的预训练权重转换脚本出现torchair链接失败问题
收藏回复举报
910A上运行Modellink中llama2-7B的预训练权重转换脚本出现torchair链接失败问题
t('forum.solved') 已解决
发表于2024-10-15 16:34:29
0 查看

运行环境:torch=2.1.0,torch_npu=2.1.0.post3,cann=8.0.RC1

问题描述:尝试在910上完成llama2-7B的预训练和推理部署,参考modelLink使用指南文档:examples/README.md · Ascend/ModelLink - Gitee.com选择了llama2-7B大模型熟悉适配流程

运行Huggingface权重转换到Megatron-LM格式的脚本bash examples/llama2/ckpt_convert_llama2_hf2legacy.sh时出现了如下报错:

ImportError: /home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/core/_abi_compat_ge_apis.so: undefined symbol: _ZN2ge5Graph28LoadFromSerializedModelArrayEPKvm 。

在gitee/torchair的issues下看到也有同样问题,下面给出的解决方法是:将CANN升级到8.0版本(我所使用的910A上预设CANN版本是7.0.RC1的)

可是我升级到CANN8.0.RC1并重新安装开发环境后,仍旧存在该报错。详细信息如下:

(fbig) [ma-user ModelLink]$bash examples/llama2/ckpt_convert_llama2_hf2legacy.sh 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/utils/path_manager.py:79: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current user. 

  warnings.warn(f"Warning: The {path} owner does not match the current user.") 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/utils/path_manager.py:79: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/7.0.0.alpha001/aarch64-linux/ascend_toolkit_install.info owner does not match the current user. 

  warnings.warn(f"Warning: The {path} owner does not match the current user.") 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:211: ImportWarning: 

    ************************************************************************************************************* 

    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now.. 

    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now.. 

    The backend in torch.distributed.init_process_group set to hccl now.. 

    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now.. 

    The device parameters have been replaced with npu in the function below: 

    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty 

    ************************************************************************************************************* 

  warnings.warn(msg, ImportWarning) 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:171: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it. 

  warnings.warn(msg, RuntimeWarning) 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:171: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it. 

  warnings.warn(msg, RuntimeWarning) 

Traceback (most recent call last): 

  File "convert_ckpt.py", line 23, in <module> 

    import modellink 

  File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/__init__.py", line 24, in <module> 

    from . import patchs 

  File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/__init__.py", line 21, in <module> 

    exec_adaptation() 

  File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 69, in exec_adaptation 

    patch_megatron_core() 

  File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 93, in patch_megatron_core 

    patch_core_models(modellink_args) 

  File "/home/ma-user/work/sp/bigmodle/ModelLink/modellink/patchs/megatron_patch.py", line 129, in patch_core_models 

    from mindspeed.core.models.common.embeddings.rotary_pos_embedding import get_pos_emb_on_this_cp_rank 

  File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/core/models/common/embeddings/rotary_pos_embedding.py", line 12, in <module> 

    from mindspeed.ops.npu_rotary_position_embedding import npu_rotary_position_embedding 

  File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/ops/npu_rotary_position_embedding.py", line 2, in <module> 

    from mindspeed.op_builder import RotaryPositionEmbeddingOpBuilder 

  File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/__init__.py", line 11, in <module> 

    from .gmm_builder import GMMOpBuilder, GMMV2OpBuilder 

  File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/gmm_builder.py", line 18, in <module> 

    from torchair.ge_concrete_graph.fx2ge_converter import register_fx_node_ge_converter 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 17, in __getattr__ 

    raise self._e 

  File "/home/ma-user/anaconda3/envs/fbig/MindSpeed/mindspeed/op_builder/gmm_builder.py", line 12, in <module> 

    from torchair import ge 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 17, in __getattr__ 

    raise self._e 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py", line 36, in _get_default_backend 

    from . import torchair 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/__init__.py", line 5, in <module> 

    from torchair.npu_fx_compiler import get_compiler 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/__init__.py", line 5, in <module> 

    from torchair.npu_fx_compiler import get_compiler 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/npu_fx_compiler.py", line 31, in <module> 

    from torchair.ge_concrete_graph.fx2ge_converter import GeConcreteGraph as ConcreteGraph 

  File "/home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/ge_concrete_graph/fx2ge_converter.py", line 22, in <module> 

    from torchair.core import _torchair 

ImportError: /home/ma-user/anaconda3/envs/fbig/lib/python3.8/site-packages/torch_npu/dynamo/torchair/core/_abi_compat_ge_apis.so: undefined symbol: _ZN2ge5Graph28LoadFromSerializedModelArrayEPKvm 

/home/ma-user/anaconda3/envs/fbig/lib/python3.8/tempfile.py:827: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2aigp1a7'> 

  _warnings.warn(warn_message, ResourceWarning) 

请问该如何有效解决该报错。

我要发帖子