环境信息：
服务器： Atlas 800
操作系统：CentOS7.6(4.14.0-115.el7a.0.1.aarch64)
驱动：23.0.rc3
容器版本：24.0.7
问题复现步骤
步骤1：
下载容器镜像，
docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025
步骤2：
创建容器，
docker run -itd -u root \
--ipc=host \
--network host \
--device=/dev/davinci0 \
--device=/dev/davinci1 \
--device=/dev/davinci2 \
--device=/dev/davinci3 \
--device=/dev/davinci4 \
--device=/dev/davinci5 \
--device=/dev/davinci6 \
--device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /etc/localtime:/etc/localtime \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /var/log/npu/:/usr/slog \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /home/data:/home/data \
--name test-mindformers \
swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025 \
/bin/bash
步骤3：
拉取mindformers代码仓，并安装部分软件
cd /root/
git clone -b dev cid:link_0
pip install tokenizers
pip install pyarrow
步骤4：
生成rank_table_file，内容如下
(mindspore2.2_py39)root@910-ak-19:~/mindformers/research/baichuan2# cat single/hccl_2p_01_127.0.1.1.json
{
"version": "1.0",
"server_count": "1",
"server_list": [
{
"server_id": "127.0.1.1",
"device": [
{
"device_id": "0",
"device_ip": "10.20.10.1",
"rank_id": "0"
},
{
"device_id": "1",
"device_ip": "10.20.11.5",
"rank_id": "1"
}
],
"host_nic_ip": "reserve"
}
],
"status": "completed"
步骤5：
下载权重和tokenizer，下载完成后整个目录如下。
(mindspore2.2_py39) root@910-ak-19:~/mindformers/research/baichuan2# tree single/
single/
|-- hccl_2p_01_127.0.1.1.json
|-- rank_0
| `-- Baichuan2-13B-Chat.ckpt
`-- tokenizer.model
步骤6：
参考《自动权重转换》中的“推理案例三完整权重自动切分为2卡分布式权重”，进行权重转换。
https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md#%E6%8E%A8%E7%90%86%E6%A1%88%E4%BE%8B%E4%B8%89%E5%AE%8C%E6%95%B4%E6%9D%83%E9%87%8D%E8%87%AA%E5%8A%A8%E5%88%87%E5%88%86%E4%B8%BA2%E5%8D%A1%E5%88%86%E5%B8%83%E5%BC%8F%E6%9D%83%E9%87%8D
修改完的配置如下：
(mindspore2.2_py39)root@910-ak-19:~/mindformers/research/baichuan2#cat run_baichuan2_13b.yaml
seed: 0
output_dir: './output' # path to save checkpoint/strategy
load_checkpoint: '/root/mindformers/research/baichuan2/single/'
src_strategy_path_or_dir: ''
auto_trans_ckpt: True # If true, auto transform load_checkpoint to load in distributed model
only_save_strategy: False
resume_training: False
run_mode: 'predict'
# trainer config
trainer:
type: CausalLanguageModelingTrainer
model_name: 'baichuan2_13b'
# if True, do evaluate during the training process. if false, do nothing.
# note that the task trainer should support _evaluate_in_training function.
do_eval: False
# runner config
runner_config:
epochs: 1
batch_size: 2
sink_mode: True
sink_size: 2
# optimizer
optimizer:
type: FP32StateAdamWeightDecay
beta1: 0.9
beta2: 0.95
eps: 1.e-8
# lr sechdule
lr_schedule:
type: CosineWithWarmUpLR
learning_rate: 2.e-5 # pretrain：3.e-4
lr_end: 1.e-6 # pretrain：3.e-5
warmup_ratio: 0.03
total_steps: -1 # -1 means it will load the total steps of the dataset
# dataset
train_dataset: &train_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: True
input_columns: ["input_ids", "labels"] # "input_ids", "labels" , labels are used in instruction finetune.
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: True
repeat: 1
numa_enable: False
prefetch_size: 1
train_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *train_dataset
# eval dataset
eval_dataset: &eval_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: False
input_columns: ["input_ids", "labels"]
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: False
repeat: 1
numa_enable: False
prefetch_size: 1
eval_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *eval_dataset
use_parallel: True
# parallel context config
parallel:
parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel
gradients_mean: False
enable_alltoall: False
full_batch: True
search_mode: "sharding_propagation"
enable_parallel_optimizer: True
strategy_ckpt_save_file: "./ckpt_strategy.ckpt"
parallel_optimizer_config:
gradient_accumulation_shard: False
parallel_optimizer_threshold: 64
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 1
model_parallel: 2
pipeline_stage: 1
use_seq_parallel: False
micro_batch_num: 1
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
# recompute config
recompute_config:
recompute: True
select_recompute: False
parallel_optimizer_comm_recompute: False
mp_comm_recompute: True
recompute_slice_activation: True
# callbacks
callbacks:
- type: MFLossMonitor
- type: CheckpointMointor
prefix: "baichuan2_13b"
save_checkpoint_steps: 1000
keep_checkpoint_max: 5
integrated_save: False
async_save: False
- type: ObsMonitor
# mindspore context init config
context:
mode: 0 #0--Graph Mode; 1--Pynative Mode
device_target: "Ascend"
enable_graph_kernel: False
graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true"
max_call_depth: 10000
max_device_memory: "31GB"
save_graphs: False
save_graphs_path: "./graph"
device_id: 0
# model config
model:
model_config:
type: LlamaConfig
batch_size: 1 # add for increase predict
seq_length: 512
hidden_size: 5120
num_layers: 40
num_heads: 40
vocab_size: 125696
multiple_of: 128
rms_norm_eps: 1.0e-6
bos_token_id: 1
eos_token_id: 2
pad_token_id: 0
ignore_token_id: -100
compute_dtype: "float16"
layernorm_compute_type: "float32"
softmax_compute_type: "float32"
param_init_type: "float16"
use_past: True
pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2
extend_method: "None" # support "None", "PI", "NTK"
compute_in_2d: False
use_flash_attention: False
offset: 0
use_past_shard: False
checkpoint_name_or_path: "/root/mindformers/research/baichuan2/single/rank_0/Baichuan2-13B-Chat.ckpt"
repetition_penalty: 1
temperature: 1.0
max_decode_length: 512
top_k: 3
top_p: 1
do_sample: False
arch:
type: Baichuan13BV2ForCausalLM
processor:
return_tensors: ms
tokenizer:
vocab_file: "/root/mindformers/research/baichuan2/single/tokenizer.model"
unk_token: '<unk>'
bos_token: '<s>'
eos_token: '</s>'
pad_token: '<unk>'
type: Baichuan2Tokenizer
type: LlamaProcessor
# metric
metric:
type: PerplexityMetric
# wrapper cell config
runner_wrapper:
type: MFTrainOneStepCell
scale_sense:
type: DynamicLossScaleUpdateCell
loss_scale_value: 65536
scale_factor: 2
scale_window: 1000
use_clip_grad: True
eval_callbacks:
- type: ObsMonitor
auto_tune: False
filepath_prefix: './autotune'
autotune_per_step: 10
profile: False
profile_start_step: 1
profile_stop_step: 10
init_start_profile: False
profile_communication: False
profile_memory: True
layer_scale: False
layer_decay: 0.65
lr_scale_factor: 256
# aicc
remote_save_url: "Please input obs url on AICC platform."
步骤7：
进行推理转换
cd /root/mindformers/scripts/
bash run_distribute.sh /root/mindformers/research/baichuan2/single/hccl_2p_01_127.0.1.1.json /root/mindformers/research/baichuan2/run_baichuan2_13b.yaml [0,2] predict "I love beijing, because"
输出如下：
output_dir is /root/mindformers/output
start training for rank 0, device 0
log saved in /root/mindformers/output/log/rank_0
start training for rank 1, device 1
log saved in /root/mindformers/output/log/rank_1
生成的output目录内容如下：
(mindspore2.2_py39) root@910-ak-19:~/mindformers/scripts# tree /root/mindformers/output/
/root/mindformers/output/
|-- log
| |-- rank_0
| | |-- error.log
| | |-- info.log
| | `-- mindformer.log
| `-- rank_1
| |-- error.log
| |-- info.log
| `-- mindformer.log
`-- strategy
4 directories, 6 files
出现错误：
ValueError: Can't find class type models class name Baichuan13BV2ForCausalLM in class registry
Info.log 日志如下：
(mindspore2.2_py39) root@910-ak-19:~/mindformers/scripts# cat /root/mindformers/output/log/rank_0/info.log
[INFO] 2024-02-01 10:17:45,296 [mindformers/tools/utils.py:153] set_output_path: set output path to '/root/mindformers/output'
[INFO] 2024-02-01 10:17:46,841 [mindformers/scripts/mf_parallel0/run_mindformer.py:110] main: .........Build context config..........
[INFO] 2024-02-01 10:17:46,842 [mindformers/core/parallel_config.py:45] build_parallel_config: initial recompute_config from dict: {'recompute': True, 'select_recompute': False, 'parallel_optimizer_comm_recompute': False, 'mp_comm_recompute': True, 'recompute_slice_activation': True}
[INFO] 2024-02-01 10:17:46,842 [mindformers/core/parallel_config.py:51] build_parallel_config: initial parallel_config from dict: {'data_parallel': 1, 'model_parallel': 2, 'pipeline_stage': 1, 'use_seq_parallel': False, 'micro_batch_num': 1, 'vocab_emb_dp': True, 'gradient_aggregation_group': 4}
[INFO] 2024-02-01 10:17:46,843 [mindformers/scripts/mf_parallel0/run_mindformer.py:112] main: context config is: [ParallelConfig]
_recompute:[ParallelConfig]
_recompute:True
_select_recompute:False
_parallel_optimizer_comm_recompute:False
_mp_comm_recompute:True
_recompute_slice_activation:True
select_recompute:False
use_seq_parallel:False
_gradient_aggregation_group:4
_embed_dp_mp_config:[ParallelConfig]
_dp_mp_config:[ParallelConfig]
_data_parallel:1
_model_parallel:2
use_seq_parallel:False
select_recompute:False
_vocab_emb_dp:True
use_seq_parallel:False
select_recompute:False
_pp_config:[ParallelConfig]
_pipeline_stage:1
_micro_batch_num:1
_moe_config:[ParallelConfig]
_dpmp:[ParallelConfig]
_data_parallel:1
_model_parallel:2
use_seq_parallel:False
select_recompute:False
_expert_parallel:1
use_seq_parallel:False
select_recompute:False
[INFO] 2024-02-01 10:17:46,844 [mindformers/scripts/mf_parallel0/run_mindformer.py:113] main: moe config is: <mindformers.modules.transformer.moe.MoEConfig object at 0xffff1711bfd0>
[INFO] 2024-02-01 10:17:46,845 [mindformers/scripts/mf_parallel0/run_mindformer.py:47] update_checkpoint_config: Leave load_checkpoint may because:
[INFO] 2024-02-01 10:17:46,845 [mindformers/scripts/mf_parallel0/run_mindformer.py:48] update_checkpoint_config: 1. resume training need resume training info.
[INFO] 2024-02-01 10:17:46,846 [mindformers/scripts/mf_parallel0/run_mindformer.py:49] update_checkpoint_config: 2. need load distributed shard checkpoint.
[INFO] 2024-02-01 10:17:46,847 [mindformers/trainer/base_trainer.py:85] __init__: Now Running Task is: text_generation, Model is: baichuan2_13b
[INFO] 2024-02-01 10:17:46,848 [mindformers/trainer/base_trainer.py:191] _check_global_batch_size_for_auto_parallel: The current parallel mode is semi_auto_parallel, full batch is True,so global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_interleave_num * gradient_accumulation_steps = 2 = 2 * 1 * 1 * 1
[INFO] 2024-02-01 10:17:46,849 [mindformers/trainer/base_trainer.py:383] create_network: .........Build Network From Config..........
[WARNING] 2024-02-01 10:17:46,849 [mindformers/models/llama/llama_config.py:183] __init__: Argument `pretrain_seqlen` is deprecated. Use `scaling_factor` instead.
[WARNING] 2024-02-01 10:17:46,850 [mindformers/models/llama/llama_config.py:186] __init__: Argument `compute_in_2d` is deprecated.
[WARNING] 2024-02-01 10:17:46,850 [mindformers/models/llama/llama_config.py:189] __init__: Argument `use_past_shard` is deprecated.
[ERROR] 2024-02-01 10:17:46,853 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/root/mindformers/scripts/mf_parallel0/run_mindformer.py", line 144, in main
create_task_trainer(config)
File "/root/mindformers/scripts/mf_parallel0/run_mindformer.py", line 90, in create_task_trainer
trainer.predict(config, is_full_config=True, batch_size=config.predict_batch_size)
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 315, in predict
return self.predict_process(config=config,
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 838, in predict_process
network = self.create_network(
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 388, in create_network
network = build_model(self.config.model, default_args=default_args)
File "/root/mindformers/scripts/mf_parallel0/mindformers/models/build_model.py", line 62, in build_model
return MindFormerRegister.get_instance_from_cfg(
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/register/register.py", line 184, in get_instance_from_cfg
obj_cls = cls.get_cls(module_type, obj_type)
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/register/register.py", line 140, in get_cls
raise ValueError("Can't find class type {} class name {} \
ValueError: Can't find class type models class name Baichuan13BV2ForCausalLM in class registry
点赞
收藏
回复
分享
举报
已解决
环境信息：
服务器： Atlas 800
操作系统：CentOS7.6(4.14.0-115.el7a.0.1.aarch64)
驱动：23.0.rc3
容器版本：24.0.7
问题复现步骤
步骤1：
下载容器镜像，
docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025
步骤2：
创建容器，
docker run -itd -u root \
--ipc=host \
--network host \
--device=/dev/davinci0 \
--device=/dev/davinci1 \
--device=/dev/davinci2 \
--device=/dev/davinci3 \
--device=/dev/davinci4 \
--device=/dev/davinci5 \
--device=/dev/davinci6 \
--device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /etc/localtime:/etc/localtime \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /var/log/npu/:/usr/slog \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /home/data:/home/data \
--name test-mindformers \
swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025 \
/bin/bash
步骤3：
拉取mindformers代码仓，并安装部分软件
cd /root/
git clone -b dev cid:link_0
pip install tokenizers
pip install pyarrow
步骤4：
生成rank_table_file，内容如下
(mindspore2.2_py39)root@910-ak-19:~/mindformers/research/baichuan2# cat single/hccl_2p_01_127.0.1.1.json
{
"version": "1.0",
"server_count": "1",
"server_list": [
{
"server_id": "127.0.1.1",
"device": [
{
"device_id": "0",
"device_ip": "10.20.10.1",
"rank_id": "0"
},
{
"device_id": "1",
"device_ip": "10.20.11.5",
"rank_id": "1"
}
],
"host_nic_ip": "reserve"
}
],
"status": "completed"
步骤5：
下载权重和tokenizer，下载完成后整个目录如下。
(mindspore2.2_py39) root@910-ak-19:~/mindformers/research/baichuan2# tree single/
single/
|-- hccl_2p_01_127.0.1.1.json
|-- rank_0
| `-- Baichuan2-13B-Chat.ckpt
`-- tokenizer.model
步骤6：
参考《自动权重转换》中的“推理案例三完整权重自动切分为2卡分布式权重”，进行权重转换。
https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md#%E6%8E%A8%E7%90%86%E6%A1%88%E4%BE%8B%E4%B8%89%E5%AE%8C%E6%95%B4%E6%9D%83%E9%87%8D%E8%87%AA%E5%8A%A8%E5%88%87%E5%88%86%E4%B8%BA2%E5%8D%A1%E5%88%86%E5%B8%83%E5%BC%8F%E6%9D%83%E9%87%8D
修改完的配置如下：
(mindspore2.2_py39)root@910-ak-19:~/mindformers/research/baichuan2#cat run_baichuan2_13b.yaml
seed: 0
output_dir: './output' # path to save checkpoint/strategy
load_checkpoint: '/root/mindformers/research/baichuan2/single/'
src_strategy_path_or_dir: ''
auto_trans_ckpt: True # If true, auto transform load_checkpoint to load in distributed model
only_save_strategy: False
resume_training: False
run_mode: 'predict'
# trainer config
trainer:
type: CausalLanguageModelingTrainer
model_name: 'baichuan2_13b'
# if True, do evaluate during the training process. if false, do nothing.
# note that the task trainer should support _evaluate_in_training function.
do_eval: False
# runner config
runner_config:
epochs: 1
batch_size: 2
sink_mode: True
sink_size: 2
# optimizer
optimizer:
type: FP32StateAdamWeightDecay
beta1: 0.9
beta2: 0.95
eps: 1.e-8
# lr sechdule
lr_schedule:
type: CosineWithWarmUpLR
learning_rate: 2.e-5 # pretrain：3.e-4
lr_end: 1.e-6 # pretrain：3.e-5
warmup_ratio: 0.03
total_steps: -1 # -1 means it will load the total steps of the dataset
# dataset
train_dataset: &train_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: True
input_columns: ["input_ids", "labels"] # "input_ids", "labels" , labels are used in instruction finetune.
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: True
repeat: 1
numa_enable: False
prefetch_size: 1
train_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *train_dataset
# eval dataset
eval_dataset: &eval_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: False
input_columns: ["input_ids", "labels"]
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: False
repeat: 1
numa_enable: False
prefetch_size: 1
eval_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *eval_dataset
use_parallel: True
# parallel context config
parallel:
parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel
gradients_mean: False
enable_alltoall: False
full_batch: True
search_mode: "sharding_propagation"
enable_parallel_optimizer: True
strategy_ckpt_save_file: "./ckpt_strategy.ckpt"
parallel_optimizer_config:
gradient_accumulation_shard: False
parallel_optimizer_threshold: 64
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 1
model_parallel: 2
pipeline_stage: 1
use_seq_parallel: False
micro_batch_num: 1
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
# recompute config
recompute_config:
recompute: True
select_recompute: False
parallel_optimizer_comm_recompute: False
mp_comm_recompute: True
recompute_slice_activation: True
# callbacks
callbacks:
- type: MFLossMonitor
- type: CheckpointMointor
prefix: "baichuan2_13b"
save_checkpoint_steps: 1000
keep_checkpoint_max: 5
integrated_save: False
async_save: False
- type: ObsMonitor
# mindspore context init config
context:
mode: 0 #0--Graph Mode; 1--Pynative Mode
device_target: "Ascend"
enable_graph_kernel: False
graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true"
max_call_depth: 10000
max_device_memory: "31GB"
save_graphs: False
save_graphs_path: "./graph"
device_id: 0
# model config
model:
model_config:
type: LlamaConfig
batch_size: 1 # add for increase predict
seq_length: 512
hidden_size: 5120
num_layers: 40
num_heads: 40
vocab_size: 125696
multiple_of: 128
rms_norm_eps: 1.0e-6
bos_token_id: 1
eos_token_id: 2
pad_token_id: 0
ignore_token_id: -100
compute_dtype: "float16"
layernorm_compute_type: "float32"
softmax_compute_type: "float32"
param_init_type: "float16"
use_past: True
pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2
extend_method: "None" # support "None", "PI", "NTK"
compute_in_2d: False
use_flash_attention: False
offset: 0
use_past_shard: False
checkpoint_name_or_path: "/root/mindformers/research/baichuan2/single/rank_0/Baichuan2-13B-Chat.ckpt"
repetition_penalty: 1
temperature: 1.0
max_decode_length: 512
top_k: 3
top_p: 1
do_sample: False
arch:
type: Baichuan13BV2ForCausalLM
processor:
return_tensors: ms
tokenizer:
vocab_file: "/root/mindformers/research/baichuan2/single/tokenizer.model"
unk_token: '<unk>'
bos_token: '<s>'
eos_token: '</s>'
pad_token: '<unk>'
type: Baichuan2Tokenizer
type: LlamaProcessor
# metric
metric:
type: PerplexityMetric
# wrapper cell config
runner_wrapper:
type: MFTrainOneStepCell
scale_sense:
type: DynamicLossScaleUpdateCell
loss_scale_value: 65536
scale_factor: 2
scale_window: 1000
use_clip_grad: True
eval_callbacks:
- type: ObsMonitor
auto_tune: False
filepath_prefix: './autotune'
autotune_per_step: 10
profile: False
profile_start_step: 1
profile_stop_step: 10
init_start_profile: False
profile_communication: False
profile_memory: True
layer_scale: False
layer_decay: 0.65
lr_scale_factor: 256
# aicc
remote_save_url: "Please input obs url on AICC platform."
步骤7：
进行推理转换
cd /root/mindformers/scripts/
bash run_distribute.sh /root/mindformers/research/baichuan2/single/hccl_2p_01_127.0.1.1.json /root/mindformers/research/baichuan2/run_baichuan2_13b.yaml [0,2] predict "I love beijing, because"
输出如下：
output_dir is /root/mindformers/output
start training for rank 0, device 0
log saved in /root/mindformers/output/log/rank_0
start training for rank 1, device 1
log saved in /root/mindformers/output/log/rank_1
生成的output目录内容如下：
(mindspore2.2_py39) root@910-ak-19:~/mindformers/scripts# tree /root/mindformers/output/
/root/mindformers/output/
|-- log
| |-- rank_0
| | |-- error.log
| | |-- info.log
| | `-- mindformer.log
| `-- rank_1
| |-- error.log
| |-- info.log
| `-- mindformer.log
`-- strategy
4 directories, 6 files
出现错误：
ValueError: Can't find class type models class name Baichuan13BV2ForCausalLM in class registry
Info.log 日志如下：
(mindspore2.2_py39) root@910-ak-19:~/mindformers/scripts# cat /root/mindformers/output/log/rank_0/info.log
[INFO] 2024-02-01 10:17:45,296 [mindformers/tools/utils.py:153] set_output_path: set output path to '/root/mindformers/output'
[INFO] 2024-02-01 10:17:46,841 [mindformers/scripts/mf_parallel0/run_mindformer.py:110] main: .........Build context config..........
[INFO] 2024-02-01 10:17:46,842 [mindformers/core/parallel_config.py:45] build_parallel_config: initial recompute_config from dict: {'recompute': True, 'select_recompute': False, 'parallel_optimizer_comm_recompute': False, 'mp_comm_recompute': True, 'recompute_slice_activation': True}
[INFO] 2024-02-01 10:17:46,842 [mindformers/core/parallel_config.py:51] build_parallel_config: initial parallel_config from dict: {'data_parallel': 1, 'model_parallel': 2, 'pipeline_stage': 1, 'use_seq_parallel': False, 'micro_batch_num': 1, 'vocab_emb_dp': True, 'gradient_aggregation_group': 4}
[INFO] 2024-02-01 10:17:46,843 [mindformers/scripts/mf_parallel0/run_mindformer.py:112] main: context config is: [ParallelConfig]
_recompute:[ParallelConfig]
_recompute:True
_select_recompute:False
_parallel_optimizer_comm_recompute:False
_mp_comm_recompute:True
_recompute_slice_activation:True
select_recompute:False
use_seq_parallel:False
_gradient_aggregation_group:4
_embed_dp_mp_config:[ParallelConfig]
_dp_mp_config:[ParallelConfig]
_data_parallel:1
_model_parallel:2
use_seq_parallel:False
select_recompute:False
_vocab_emb_dp:True
use_seq_parallel:False
select_recompute:False
_pp_config:[ParallelConfig]
_pipeline_stage:1
_micro_batch_num:1
_moe_config:[ParallelConfig]
_dpmp:[ParallelConfig]
_data_parallel:1
_model_parallel:2
use_seq_parallel:False
select_recompute:False
_expert_parallel:1
use_seq_parallel:False
select_recompute:False
[INFO] 2024-02-01 10:17:46,844 [mindformers/scripts/mf_parallel0/run_mindformer.py:113] main: moe config is: <mindformers.modules.transformer.moe.MoEConfig object at 0xffff1711bfd0>
[INFO] 2024-02-01 10:17:46,845 [mindformers/scripts/mf_parallel0/run_mindformer.py:47] update_checkpoint_config: Leave load_checkpoint may because:
[INFO] 2024-02-01 10:17:46,845 [mindformers/scripts/mf_parallel0/run_mindformer.py:48] update_checkpoint_config: 1. resume training need resume training info.
[INFO] 2024-02-01 10:17:46,846 [mindformers/scripts/mf_parallel0/run_mindformer.py:49] update_checkpoint_config: 2. need load distributed shard checkpoint.
[INFO] 2024-02-01 10:17:46,847 [mindformers/trainer/base_trainer.py:85] __init__: Now Running Task is: text_generation, Model is: baichuan2_13b
[INFO] 2024-02-01 10:17:46,848 [mindformers/trainer/base_trainer.py:191] _check_global_batch_size_for_auto_parallel: The current parallel mode is semi_auto_parallel, full batch is True,so global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_interleave_num * gradient_accumulation_steps = 2 = 2 * 1 * 1 * 1
[INFO] 2024-02-01 10:17:46,849 [mindformers/trainer/base_trainer.py:383] create_network: .........Build Network From Config..........
[WARNING] 2024-02-01 10:17:46,849 [mindformers/models/llama/llama_config.py:183] __init__: Argument `pretrain_seqlen` is deprecated. Use `scaling_factor` instead.
[WARNING] 2024-02-01 10:17:46,850 [mindformers/models/llama/llama_config.py:186] __init__: Argument `compute_in_2d` is deprecated.
[WARNING] 2024-02-01 10:17:46,850 [mindformers/models/llama/llama_config.py:189] __init__: Argument `use_past_shard` is deprecated.
[ERROR] 2024-02-01 10:17:46,853 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/root/mindformers/scripts/mf_parallel0/run_mindformer.py", line 144, in main
create_task_trainer(config)
File "/root/mindformers/scripts/mf_parallel0/run_mindformer.py", line 90, in create_task_trainer
trainer.predict(config, is_full_config=True, batch_size=config.predict_batch_size)
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 315, in predict
return self.predict_process(config=config,
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 838, in predict_process
network = self.create_network(
File "/root/mindformers/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 388, in create_network
network = build_model(self.config.model, default_args=default_args)
File "/root/mindformers/scripts/mf_parallel0/mindformers/models/build_model.py", line 62, in build_model
return MindFormerRegister.get_instance_from_cfg(
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/register/register.py", line 184, in get_instance_from_cfg
obj_cls = cls.get_cls(module_type, obj_type)
File "/root/mindformers/scripts/mf_parallel0/mindformers/tools/register/register.py", line 140, in get_cls
raise ValueError("Can't find class type {} class name {} \
ValueError: Can't find class type models class name Baichuan13BV2ForCausalLM in class registry