2024-08-13 17:26:57,810 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:57,812 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:26:59,037 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:59,040 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:26:59,263 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:59,265 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:26:59,505 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:59,522 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:26:59,546 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:59,558 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:26:59,711 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:26:59,716 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:27:00,202 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:27:00,207 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:27:01,300 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint..................
2024-08-13 17:27:01,304 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt
2024-08-13 17:27:06,634 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success.
[WARNING] ME(785172:281470923008288,Process-8):2024-08-13-17:27:06.775.441 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network.
2024-08-13 17:27:06,780 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success.
[WARNING] ME(785157:281470923008288,Process-6):2024-08-13-17:27:06.916.793 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network.
2024-08-13 17:27:07,114 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success.
[WARNING] ME(785155:281470923008288,Process-4):2024-08-13-17:27:07.295.091 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network.
[WARNING] ME(785157:281470923008288,Process-6):2024-08-13-17:27:07.315.659 [mindspore/train/serialization.py:172] The type of transformer.layers.0.attention.wq.weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network.
[CRITICAL] ME(785157:281470923008288,Process-6):2024-08-13-17:27:07.336.000 [mindspore/train/serialization.py:121] Failed to combine the net and the parameters for param transformer.layers.0.attention.wq.bias.
[WARNING] ME(785172:281470923008288,Process-8):2024-08-13-17:27:07.381.189 [mindspore/train/serialization.py:172] The type of transformer.layers.0.attention.wq.weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network.
[CRITICAL] ME(785172:281470923008288,Process-8):2024-08-13-17:27:07.393.400 [mindspore/train/serialization.py:121] Failed to combine the net and the parameters for param transformer.layers.0.attention.wq.bias.
Process Process-6:
Traceback (most recent call last):
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/data/hljzh/mindformers/chat_web/predict_process_qwen.py", line 105, in generate_process
transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 361, in transform_and_load_checkpoint
load_ckpt(config, network, optimizer=optimizer)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 755, in load_ckpt
not_load_network_params = load_param_into_net(network, checkpoint_dict)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1303, in load_param_into_net
_update_param(param, new_param, strict_load)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 127, in _update_param
raise RuntimeError(msg)
RuntimeError: For 'load_param_into_net', transformer.layers.0.attention.wq.bias in the argument 'net' should have the same shape as transformer.layers.0.attention.wq.bias in the argument 'parameter_dict'. But got its shape (512,) in the argument 'net' and shape (2048,) in the argument 'parameter_dict'.May you need to check whether the checkpoint you loaded is correct or the batch size and so on in the 'net' and 'parameter_dict' are same.
Process Process-8:
Traceback (most recent call last):
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/data/hljzh/mindformers/chat_web/predict_process_qwen.py", line 105, in generate_process
transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 361, in transform_and_load_checkpoint
load_ckpt(config, network, optimizer=optimizer)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 755, in load_ckpt
not_load_network_params = load_param_into_net(network, checkpoint_dict)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1303, in load_param_into_net
_update_param(param, new_param, strict_load)
File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 127, in _update_param
raise RuntimeError(msg)
RuntimeError: For 'load_param_into_net', transformer.layers.0.attention.wq.bias in the argument 'net' should have the same shape as transformer.layers.0.attention.wq.bias in the argument 'parameter_dict'. But got its shape (512,) in the argument 'net' and shape (2048,) in the argument 'parameter_dict'.May you need to check whether the checkpoint you loaded is correct or the batch size and so on in the 'net' and 'parameter_dict' are same.
点赞
收藏
回复
分享
举报
已解决
使用设备：800TA2
mindspore：2.2.0
mindformers：r1.0
glm3-6b和qwen-7b报错一样，但是使用单卡推理可正常对话，使用八卡分布式权重会报错，是目前不支持使用微调后的分布式权重进行chat_web对话推理吗？
进行chat_web的predict_qwen_7b.yaml内容如下：
seed: 0 output_dir: './output' # path to save checkpoint/strategy load_checkpoint: '/home/data/hljzh/qwen-result/output/transformed_checkpoint/model_dir/' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False use_parallel: True run_mode: 'predict' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'qwen_7b' # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "/home/data/gyw/mindformers_r1.0/research/qwen/alpaca.mindrecord" shuffle: True input_columns: ["input_ids", "labels", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 6500 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 65536 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-6 weight_decay: 0.1 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "qwen" save_checkpoint_steps: 10000 keep_checkpoint_max: 3 integrated_save: False async_save: False - type: ObsMonitor # default parallel of device num = 8 for Atlas 800T A2 parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 1 micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 2 # recompute config recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: False recompute_slice_activation: False model: model_config: type: QwenConfig batch_size: 1 seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 151936 intermediate_size: 11008 rms_norm_eps: 1.0e-6 emb_dropout_prob: 0.0 eos_token_id: 151643 pad_token_id: 151643 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: True use_flash_attention: False use_paged_attention: False # only supported in mslite inference block_size: 32 num_blocks: 128 is_dynamic: False use_kvcache_op: False offset: 0 checkpoint_name_or_path: "" repetition_penalty: 1 max_decode_length: 512 top_k: 0 top_p: 0.8 do_sample: False # configuration items copied from Qwen rotary_pct: 1.0 rotary_emb_base: 10000 kv_channels: 128 arch: type: QwenForCausalLM processor: return_tensors: ms tokenizer: model_max_length: 8192 vocab_file: "/home/data/hljzh/qwen-weight/qwen.tiktoken" pad_token: "<|endoftext|>" type: QwenTokenizer type: QwenProcessor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" ascend_config: precision_mode: "must_keep_origin_dtype" max_call_depth: 10000 max_device_memory: "60GB" save_graphs: False save_graphs_path: "./graph" device_id: 0,1,2,3,4,5,6,7 # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_config: save_file: "./ckpt_strategy.ckpt" only_trainable_params: False parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # aicc remote_save_url: "Please input obs url on AICC platform."拉起服务时，报错如下：
2024-08-13 17:26:57,810 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:57,812 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:26:59,037 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:59,040 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:26:59,263 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:59,265 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:26:59,505 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:59,522 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:26:59,546 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:59,558 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:26:59,711 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:26:59,716 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:27:00,202 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:27:00,207 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:27:01,300 - mindformers[mindformers/trainer/utils.py:722] - INFO - .............Start load checkpoint from checkpoint.................. 2024-08-13 17:27:01,304 - mindformers[mindformers/trainer/utils.py:249] - INFO - When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/**.ckpt 2024-08-13 17:27:06,634 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success. [WARNING] ME(785172:281470923008288,Process-8):2024-08-13-17:27:06.775.441 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network. 2024-08-13 17:27:06,780 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success. [WARNING] ME(785157:281470923008288,Process-6):2024-08-13-17:27:06.916.793 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network. 2024-08-13 17:27:07,114 - mindformers[mindformers/trainer/utils.py:262] - INFO - Distribute load is success. [WARNING] ME(785155:281470923008288,Process-4):2024-08-13-17:27:07.295.091 [mindspore/train/serialization.py:172] The type of transformer.wte.embedding_weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network. [WARNING] ME(785157:281470923008288,Process-6):2024-08-13-17:27:07.315.659 [mindspore/train/serialization.py:172] The type of transformer.layers.0.attention.wq.weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network. [CRITICAL] ME(785157:281470923008288,Process-6):2024-08-13-17:27:07.336.000 [mindspore/train/serialization.py:121] Failed to combine the net and the parameters for param transformer.layers.0.attention.wq.bias. [WARNING] ME(785172:281470923008288,Process-8):2024-08-13-17:27:07.381.189 [mindspore/train/serialization.py:172] The type of transformer.layers.0.attention.wq.weight:Float32 in 'parameter_dict' is different from the type of it in 'net':Float16, then the type convert from Float32 to Float16 in the network. [CRITICAL] ME(785172:281470923008288,Process-8):2024-08-13-17:27:07.393.400 [mindspore/train/serialization.py:121] Failed to combine the net and the parameters for param transformer.layers.0.attention.wq.bias. Process Process-6: Traceback (most recent call last): File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/data/hljzh/mindformers/chat_web/predict_process_qwen.py", line 105, in generate_process transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 361, in transform_and_load_checkpoint load_ckpt(config, network, optimizer=optimizer) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 755, in load_ckpt not_load_network_params = load_param_into_net(network, checkpoint_dict) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1303, in load_param_into_net _update_param(param, new_param, strict_load) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 127, in _update_param raise RuntimeError(msg) RuntimeError: For 'load_param_into_net', transformer.layers.0.attention.wq.bias in the argument 'net' should have the same shape as transformer.layers.0.attention.wq.bias in the argument 'parameter_dict'. But got its shape (512,) in the argument 'net' and shape (2048,) in the argument 'parameter_dict'.May you need to check whether the checkpoint you loaded is correct or the batch size and so on in the 'net' and 'parameter_dict' are same. Process Process-8: Traceback (most recent call last): File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/data/hljzh/mindformers/chat_web/predict_process_qwen.py", line 105, in generate_process transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 361, in transform_and_load_checkpoint load_ckpt(config, network, optimizer=optimizer) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 755, in load_ckpt not_load_network_params = load_param_into_net(network, checkpoint_dict) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 1303, in load_param_into_net _update_param(param, new_param, strict_load) File "/root/miniconda3/envs/mindspore2.2_py39/lib/python3.9/site-packages/mindspore/train/serialization.py", line 127, in _update_param raise RuntimeError(msg) RuntimeError: For 'load_param_into_net', transformer.layers.0.attention.wq.bias in the argument 'net' should have the same shape as transformer.layers.0.attention.wq.bias in the argument 'parameter_dict'. But got its shape (512,) in the argument 'net' and shape (2048,) in the argument 'parameter_dict'.May you need to check whether the checkpoint you loaded is correct or the batch size and so on in the 'net' and 'parameter_dict' are same.