在训练baichuan2,不进行lora下直接推理tokenizer.model文件报错
收藏回复举报
在训练baichuan2,不进行lora下直接推理tokenizer.model文件报错
t('forum.solved') 已解决
发表于2024-03-05 16:53:52
0 查看

报错如下述:所使用的文件均是从官网中下载的,今天想重新从头跑一遍模型对比效果发现报了如下的错。

2024-03-05 16:43:29,929 - mindformers[mindformers/models/modeling_utils.py:1413] - INFO - weights in /home/ma-user/work/models/Baichuan2_7B_Chat.ckpt are loaded 

Traceback (most recent call last): 

  File "/home/ma-user/work/mindformers/research/baichuan2/run_baichuan2_pipeline.py", line 34, in <module> 

    tokenizer = Baichuan2Tokenizer( 

  File "/home/ma-user/work/mindformers/research/baichuan2/baichuan2_tokenizer.py", line 94, in __init__ 

    super().__init__( 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 426, in __init__ 

    self._add_tokens( 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 545, in _add_tokens 

    if not token.special and token.normalized and getattr(self, "do_lower_case", False): 

AttributeError: 'tokenizers.AddedToken' object has no attribute 'special'

我要发帖子