报错如下述:所使用的文件均是从官网中下载的,今天想重新从头跑一遍模型对比效果发现报了如下的错。
2024-03-05 16:43:29,929 - mindformers[mindformers/models/modeling_utils.py:1413] - INFO - weights in /home/ma-user/work/models/Baichuan2_7B_Chat.ckpt are loaded
Traceback (most recent call last):
File "/home/ma-user/work/mindformers/research/baichuan2/run_baichuan2_pipeline.py", line 34, in <module>
tokenizer = Baichuan2Tokenizer(
File "/home/ma-user/work/mindformers/research/baichuan2/baichuan2_tokenizer.py", line 94, in __init__
super().__init__(
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 426, in __init__
self._add_tokens(
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 545, in _add_tokens
if not token.special and token.normalized and getattr(self, "do_lower_case", False):
AttributeError: 'tokenizers.AddedToken' object has no attribute 'special'
报错如下述:所使用的文件均是从官网中下载的,今天想重新从头跑一遍模型对比效果发现报了如下的错。
2024-03-05 16:43:29,929 - mindformers[mindformers/models/modeling_utils.py:1413] - INFO - weights in /home/ma-user/work/models/Baichuan2_7B_Chat.ckpt are loaded
Traceback (most recent call last):
File "/home/ma-user/work/mindformers/research/baichuan2/run_baichuan2_pipeline.py", line 34, in <module>
tokenizer = Baichuan2Tokenizer(
File "/home/ma-user/work/mindformers/research/baichuan2/baichuan2_tokenizer.py", line 94, in __init__
super().__init__(
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 426, in __init__
self._add_tokens(
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindformers/models/tokenization_utils.py", line 545, in _add_tokens
if not token.special and token.normalized and getattr(self, "do_lower_case", False):
AttributeError: 'tokenizers.AddedToken' object has no attribute 'special'