(chatglm_py39) [sy_cmsr@zq01-wtd-test-AIGC-0002 Baichuan2-13b]$ bash /home/sy_cmsr/work/Baichuan2-13b/run_singlenode.sh "python3.9 /home/sy_cmsr/work/Baichuan2-13b/run_baichuan2_generate.py" /home/sy_cmsr/work/Baichuan2-13b/hccl_8p_01234567_10.125.80.11.json [0,8] 8
start training for rank 0, device 0
start training for rank 1, device 1
start training for rank 2, device 2
start training for rank 3, device 3
start training for rank 4, device 4
start training for rank 5, device 5
start training for rank 6, device 6
start training for rank 7, device 7
用上述命令分8张卡跑,配置文件也照着案例写的,生成了output文件夹,为什么会报下面这个错误呢,transformed_checkpoint里面也没有东西,推理也没有结果?
Traceback (most recent call last):
File "/home/sy_cmsr/work/Baichuan2-13b/run_baichuan2_generate.py", line 64, in <module>
transform_and_load_checkpoint(baichuan2_config, baichuan2_model, baichuan2_network, infer_data, do_predict=True)
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 328, in transform_and_load_checkpoint
transform_ckpt(config, src_ckpt_strategy=src_ckpt_strategy, dst_ckpt_strategy=dst_ckpt_strategy)
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 460, in transform_ckpt
ms.transform_checkpoints(config.load_checkpoint,
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindspore/parallel/checkpoint_transform.py", line 302, in transform_checkpoints
ckpt_dict = ms.load_checkpoint(all_checkpoint_files_map.get(int(needed_rank)))
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1097, in load_checkpoint
raise ValueError(f"The loaded parameter dict is empty after filter or specify, please check whether "
ValueError: The loaded parameter dict is empty after filter or specify, please check whether 'filter_prefix' or 'specify_prefix' are set correctly.
start training for rank 0, device 0
start training for rank 1, device 1
start training for rank 2, device 2
start training for rank 3, device 3
start training for rank 4, device 4
start training for rank 5, device 5
start training for rank 6, device 6
start training for rank 7, device 7
用上述命令分8张卡跑,配置文件也照着案例写的,生成了output文件夹,为什么会报下面这个错误呢,transformed_checkpoint里面也没有东西,推理也没有结果?
Traceback (most recent call last):
File "/home/sy_cmsr/work/Baichuan2-13b/run_baichuan2_generate.py", line 64, in <module>
transform_and_load_checkpoint(baichuan2_config, baichuan2_model, baichuan2_network, infer_data, do_predict=True)
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 328, in transform_and_load_checkpoint
transform_ckpt(config, src_ckpt_strategy=src_ckpt_strategy, dst_ckpt_strategy=dst_ckpt_strategy)
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 460, in transform_ckpt
ms.transform_checkpoints(config.load_checkpoint,
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindspore/parallel/checkpoint_transform.py", line 302, in transform_checkpoints
ckpt_dict = ms.load_checkpoint(all_checkpoint_files_map.get(int(needed_rank)))
File "/home/sy_cmsr/miniconda3/envs/chatglm_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1097, in load_checkpoint
raise ValueError(f"The loaded parameter dict is empty after filter or specify, please check whether "
ValueError: The loaded parameter dict is empty after filter or specify, please check whether 'filter_prefix' or 'specify_prefix' are set correctly.