1. 通过convert文件将llama3-8B转成单个cpkt文件
2. bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict"
yaml文件没改过用默认的，就添加了checkpoint_name_or_path和vocab_file
```
seed: 0
output_dir: './output' # path to save checkpoint/strategy
load_checkpoint: ''
src_strategy_path_or_dir: ''
auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model
only_save_strategy: False
resume_training: False
run_mode: 'predict'
# trainer config
trainer:
type: CausalLanguageModelingTrainer
model_name: 'llama3_8b'
# runner config
runner_config:
epochs: 2
batch_size: 1
sink_mode: True
sink_size: 2
# eval dataset
eval_dataset: &eval_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: False
input_columns: ["input_ids"]
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: False
repeat: 1
numa_enable: False
prefetch_size: 1
eval_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *eval_dataset
use_parallel: False
# parallel context config
parallel:
parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel
gradients_mean: False
enable_alltoall: False
full_batch: True
search_mode: "sharding_propagation"
enable_parallel_optimizer: False
strategy_ckpt_save_file: "./ckpt_strategy.ckpt"
parallel_optimizer_config:
gradient_accumulation_shard: False
parallel_optimizer_threshold: 64
# default parallel of device num = 8 for Atlas 800T A2
parallel_config:
data_parallel: 1
model_parallel: 1
pipeline_stage: 1
use_seq_parallel: False
micro_batch_num: 1
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
# mindspore context init config
context:
mode: 0 #0--Graph Mode; 1--Pynative Mode
device_target: "Ascend"
enable_graph_kernel: False
max_call_depth: 10000
max_device_memory: "58GB"
save_graphs: False
save_graphs_path: "./graph"
device_id: 0
# model config
model:
model_config:
type: LlamaConfig
batch_size: 1 # add for increase predict
seq_length: 512
hidden_size: 4096
num_layers: 32
num_heads: 32
n_kv_heads: 8
vocab_size: 128256
intermediate_size: 14336
rms_norm_eps: 1.0e-5
bos_token_id: 128000
eos_token_id: 128001
pad_token_id: 128002
ignore_token_id: -100
compute_dtype: "bfloat16"
layernorm_compute_type: "float32"
softmax_compute_type: "float32"
rotary_dtype: "bfloat16"
param_init_type: "float16"
use_past: True
is_dynamic: True
scaling_factor: 1.0
theta: 500000
extend_method: "None" # support "None", "PI", "NTK"
use_flash_attention: True # FA can accelerate training or finetune
offset: 0
fine_grain_interleave: 1
checkpoint_name_or_path: "/home/linweibin/liujian/project/policy-model/mindformers/model/llama3.ckpt"
repetition_penalty: 1
max_decode_length: 512
block_size: 16
num_blocks: 512
top_k: 3
top_p: 1
do_sample: False
arch:
type: LlamaForCausalLM
processor:
return_tensors: ms
tokenizer:
model_max_length: 8192
vocab_file: "/home/linweibin/liujian/model/llama3-8B/original/tokenizer.model"
pad_token: "<|reserved_special_token_0|>"
type: Llama3Tokenizer
type: LlamaProcessor
# metric
metric:
type: PerplexityMetric
# wrapper cell config
runner_wrapper:
type: MFTrainOneStepCell
scale_sense: 1.0
use_clip_grad: True
eval_callbacks:
- type: ObsMonitor
auto_tune: False
filepath_prefix: './autotune'
autotune_per_step: 10
profile: False
profile_start_step: 4
profile_stop_step: 8
init_start_profile: False
profile_communication: False
profile_memory: True
layer_scale: False
layer_decay: 0.65
lr_scale_factor: 256
# aicc
remote_save_url: "Please input obs url on AICC platform."
```
报错如下：
```
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
2024-09-02 16:44:16,860 - mindformers[mindformers/run_mindformer.py:223] - INFO - dataset by config is used as input_data.
2024-09-02 16:44:16,861 - mindformers[mindformers/tools/utils.py:160] - INFO - set output path to '/home/linweibin/liujian/project/policy-model/mindformers/output'
2024-09-02 16:44:16,881 - mindformers[mindformers/trainer/trainer.py:952] - INFO - Load configs in /home/linweibin/liujian/project/policy-model/mindformers/configs/general/run_general_task.yaml to build trainer.
2024-09-02 16:44:16,881 - mindformers[mindformers/trainer/trainer.py:1035] - INFO - ..........Init Config..........
2024-09-02 16:44:16,881 - mindformers[mindformers/core/parallel_config.py:51] - INFO - initial parallel_config from dict: {'data_parallel': 1, 'model_parallel': 1, 'pipeline_stage': 1, 'use_seq_parallel': False, 'micro_batch_num': 1, 'vocab_emb_dp': True, 'gradient_aggregation_group': 4}
2024-09-02 16:44:16,882 - mindformers[mindformers/tools/utils.py:160] - INFO - set output path to '/home/linweibin/liujian/project/policy-model/mindformers/output'
2024-09-02 16:44:16,882 - mindformers[mindformers/tools/utils.py:175] - INFO - set strategy path to './output/strategy/ckpt_strategy_rank_0.ckpt'
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:89] - INFO - Now Running Task is: text_generation, Model is: llama3_8b
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:115] - WARNING - Input model name is not in the supported list or unspecified.
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:116] - WARNING - See the list of supported task and model name: ['baichuan2_13b', 'baichuan2_7b', 'bloom_176b', 'bloom_560m', 'bloom_65b', 'bloom_7.1b', 'codegeex2_6b', 'codellama_34b', 'common', 'deepseek_33b', 'glm2_6b', 'glm2_6b_lora', 'glm3_6b', 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat', 'gpt2', 'gpt2_13b', 'gpt2_52b', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'internlm_7b', 'internlm_7b_lora', 'llama2_13b', 'llama2_70b', 'llama2_7b', 'llama_13b', 'llama_65b', 'llama_7b', 'llama_7b_lora', 'pangualpha_13b', 'pangualpha_2_6b', 'qwen_7b', 'qwen_7b_lora', 'yi_34b', 'yi_6b']
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:117] - WARNING - The default model config: /home/linweibin/liujian/project/policy-model/mindformers/configs/gpt2/run_gpt2.yaml will now be used for the text_generation task
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:1100] - INFO - ..........Init Model..........
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:311] - INFO - ==========Trainer Init Success!==========
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:1100] - INFO - ..........Init Model..........
2024-09-02 16:44:16,888 - mindformers[mindformers/tools/cloud_adapter/cloud_monitor.py:43] - ERROR - Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/engine/validators.py", line 1015, in new_method
_, param_dict = parse_user_args(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/core/validator_helpers.py", line 472, in parse_user_args
ba = sig.bind(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3186, in bind
return self._bind(args, kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3101, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'dataset_files'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 45, in main
trainer.predict(predict_checkpoint=config.load_checkpoint, input_data=config.input_data,
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
return func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/trainer/trainer.py", line 722, in predict
input_data = build_dataset_loader(self.config.eval_dataset.data_loader)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/dataset/dataloader/build_dataloader.py", line 52, in build_dataset_loader
return MindFormerRegister.get_instance_from_cfg(
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
TypeError: MindDataset: missing a required argument: 'dataset_files'
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/engine/validators.py", line 1015, in new_method
_, param_dict = parse_user_args(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/core/validator_helpers.py", line 472, in parse_user_args
ba = sig.bind(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3186, in bind
return self._bind(args, kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3101, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'dataset_files'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 271, in <module>
main(config_)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 44, in wrapper
raise exc
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 45, in main
trainer.predict(predict_checkpoint=config.load_checkpoint, input_data=config.input_data,
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
return func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/trainer/trainer.py", line 722, in predict
input_data = build_dataset_loader(self.config.eval_dataset.data_loader)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/dataset/dataloader/build_dataloader.py", line 52, in build_dataset_loader
return MindFormerRegister.get_instance_from_cfg(
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
TypeError: MindDataset: missing a required argument: 'dataset_files'
```
dataset_files是干嘛的，推理的话要数据集嘛。
```
(policy) linweibin@user:~/liujian/project/policy-model/mindformers$ bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict"
No parameter is entered. Notice that the program will run on default 8 cards.
scripts/msrun_launcher.sh: line 119: ulimit: max user processes: cannot modify limit: Operation not permitted
Running Command: msrun --worker_num=8 --local_worker_num=8 --master_port=8118 --log_dir=output/msrun_log --join=False --cluster_time_out=600 run_mindformer.py --config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml --run_mode predict
Please check log files in output/msrun_log
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.870.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:0, log file:output/msrun_log/worker_0.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.878.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:1, log file:output/msrun_log/worker_1.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.884.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:2, log file:output/msrun_log/worker_2.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.889.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:3, log file:output/msrun_log/worker_3.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.894.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:4, log file:output/msrun_log/worker_4.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.898.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:5, log file:output/msrun_log/worker_5.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.903.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:6, log file:output/msrun_log/worker_6.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.908.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:7, log file:output/msrun_log/worker_7.log. Environment variable [RANK_ID] is exported.
(policy) linweibin@user:~/liujian/project/policy-model/mindformers$ bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict" 1
scripts/msrun_launcher.sh: line 119: ulimit: max user processes: cannot modify limit: Operation not permitted
Running Command: msrun --worker_num=1 --local_worker_num=1 --master_port=8118 --log_dir=output/msrun_log --join=False --cluster_time_out=600 run_mindformer.py --config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml --run_mode predict
Please check log files in output/msrun_log
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
[WARNING] ME(1520522:281473071022112,MainProcess):2024-09-02-16:44:04.220.00 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:0, log file:output/msrun_log/worker_0.log. Environment variable [RANK_ID] is exported.
```
点赞
收藏
回复
分享
举报
已解决
1. 通过convert文件将llama3-8B转成单个cpkt文件
2. bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict"
yaml文件没改过用默认的，就添加了checkpoint_name_or_path和vocab_file
```
seed: 0
output_dir: './output' # path to save checkpoint/strategy
load_checkpoint: ''
src_strategy_path_or_dir: ''
auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model
only_save_strategy: False
resume_training: False
run_mode: 'predict'
# trainer config
trainer:
type: CausalLanguageModelingTrainer
model_name: 'llama3_8b'
# runner config
runner_config:
epochs: 2
batch_size: 1
sink_mode: True
sink_size: 2
# eval dataset
eval_dataset: &eval_dataset
data_loader:
type: MindDataset
dataset_dir: ""
shuffle: False
input_columns: ["input_ids"]
num_parallel_workers: 8
python_multiprocessing: False
drop_remainder: False
repeat: 1
numa_enable: False
prefetch_size: 1
eval_dataset_task:
type: CausalLanguageModelDataset
dataset_config: *eval_dataset
use_parallel: False
# parallel context config
parallel:
parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel
gradients_mean: False
enable_alltoall: False
full_batch: True
search_mode: "sharding_propagation"
enable_parallel_optimizer: False
strategy_ckpt_save_file: "./ckpt_strategy.ckpt"
parallel_optimizer_config:
gradient_accumulation_shard: False
parallel_optimizer_threshold: 64
# default parallel of device num = 8 for Atlas 800T A2
parallel_config:
data_parallel: 1
model_parallel: 1
pipeline_stage: 1
use_seq_parallel: False
micro_batch_num: 1
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
# mindspore context init config
context:
mode: 0 #0--Graph Mode; 1--Pynative Mode
device_target: "Ascend"
enable_graph_kernel: False
max_call_depth: 10000
max_device_memory: "58GB"
save_graphs: False
save_graphs_path: "./graph"
device_id: 0
# model config
model:
model_config:
type: LlamaConfig
batch_size: 1 # add for increase predict
seq_length: 512
hidden_size: 4096
num_layers: 32
num_heads: 32
n_kv_heads: 8
vocab_size: 128256
intermediate_size: 14336
rms_norm_eps: 1.0e-5
bos_token_id: 128000
eos_token_id: 128001
pad_token_id: 128002
ignore_token_id: -100
compute_dtype: "bfloat16"
layernorm_compute_type: "float32"
softmax_compute_type: "float32"
rotary_dtype: "bfloat16"
param_init_type: "float16"
use_past: True
is_dynamic: True
scaling_factor: 1.0
theta: 500000
extend_method: "None" # support "None", "PI", "NTK"
use_flash_attention: True # FA can accelerate training or finetune
offset: 0
fine_grain_interleave: 1
checkpoint_name_or_path: "/home/linweibin/liujian/project/policy-model/mindformers/model/llama3.ckpt"
repetition_penalty: 1
max_decode_length: 512
block_size: 16
num_blocks: 512
top_k: 3
top_p: 1
do_sample: False
arch:
type: LlamaForCausalLM
processor:
return_tensors: ms
tokenizer:
model_max_length: 8192
vocab_file: "/home/linweibin/liujian/model/llama3-8B/original/tokenizer.model"
pad_token: "<|reserved_special_token_0|>"
type: Llama3Tokenizer
type: LlamaProcessor
# metric
metric:
type: PerplexityMetric
# wrapper cell config
runner_wrapper:
type: MFTrainOneStepCell
scale_sense: 1.0
use_clip_grad: True
eval_callbacks:
- type: ObsMonitor
auto_tune: False
filepath_prefix: './autotune'
autotune_per_step: 10
profile: False
profile_start_step: 4
profile_stop_step: 8
init_start_profile: False
profile_communication: False
profile_memory: True
layer_scale: False
layer_decay: 0.65
lr_scale_factor: 256
# aicc
remote_save_url: "Please input obs url on AICC platform."
```
报错如下：
```
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
2024-09-02 16:44:16,860 - mindformers[mindformers/run_mindformer.py:223] - INFO - dataset by config is used as input_data.
2024-09-02 16:44:16,861 - mindformers[mindformers/tools/utils.py:160] - INFO - set output path to '/home/linweibin/liujian/project/policy-model/mindformers/output'
2024-09-02 16:44:16,881 - mindformers[mindformers/trainer/trainer.py:952] - INFO - Load configs in /home/linweibin/liujian/project/policy-model/mindformers/configs/general/run_general_task.yaml to build trainer.
2024-09-02 16:44:16,881 - mindformers[mindformers/trainer/trainer.py:1035] - INFO - ..........Init Config..........
2024-09-02 16:44:16,881 - mindformers[mindformers/core/parallel_config.py:51] - INFO - initial parallel_config from dict: {'data_parallel': 1, 'model_parallel': 1, 'pipeline_stage': 1, 'use_seq_parallel': False, 'micro_batch_num': 1, 'vocab_emb_dp': True, 'gradient_aggregation_group': 4}
2024-09-02 16:44:16,882 - mindformers[mindformers/tools/utils.py:160] - INFO - set output path to '/home/linweibin/liujian/project/policy-model/mindformers/output'
2024-09-02 16:44:16,882 - mindformers[mindformers/tools/utils.py:175] - INFO - set strategy path to './output/strategy/ckpt_strategy_rank_0.ckpt'
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:89] - INFO - Now Running Task is: text_generation, Model is: llama3_8b
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:115] - WARNING - Input model name is not in the supported list or unspecified.
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:116] - WARNING - See the list of supported task and model name: ['baichuan2_13b', 'baichuan2_7b', 'bloom_176b', 'bloom_560m', 'bloom_65b', 'bloom_7.1b', 'codegeex2_6b', 'codellama_34b', 'common', 'deepseek_33b', 'glm2_6b', 'glm2_6b_lora', 'glm3_6b', 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat', 'gpt2', 'gpt2_13b', 'gpt2_52b', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'internlm_7b', 'internlm_7b_lora', 'llama2_13b', 'llama2_70b', 'llama2_7b', 'llama_13b', 'llama_65b', 'llama_7b', 'llama_7b_lora', 'pangualpha_13b', 'pangualpha_2_6b', 'qwen_7b', 'qwen_7b_lora', 'yi_34b', 'yi_6b']
2024-09-02 16:44:16,883 - mindformers[mindformers/trainer/base_trainer.py:117] - WARNING - The default model config: /home/linweibin/liujian/project/policy-model/mindformers/configs/gpt2/run_gpt2.yaml will now be used for the text_generation task
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:1100] - INFO - ..........Init Model..........
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:311] - INFO - ==========Trainer Init Success!==========
2024-09-02 16:44:16,884 - mindformers[mindformers/trainer/trainer.py:1100] - INFO - ..........Init Model..........
2024-09-02 16:44:16,888 - mindformers[mindformers/tools/cloud_adapter/cloud_monitor.py:43] - ERROR - Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/engine/validators.py", line 1015, in new_method
_, param_dict = parse_user_args(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/core/validator_helpers.py", line 472, in parse_user_args
ba = sig.bind(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3186, in bind
return self._bind(args, kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3101, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'dataset_files'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 45, in main
trainer.predict(predict_checkpoint=config.load_checkpoint, input_data=config.input_data,
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
return func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/trainer/trainer.py", line 722, in predict
input_data = build_dataset_loader(self.config.eval_dataset.data_loader)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/dataset/dataloader/build_dataloader.py", line 52, in build_dataset_loader
return MindFormerRegister.get_instance_from_cfg(
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
TypeError: MindDataset: missing a required argument: 'dataset_files'
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/engine/validators.py", line 1015, in new_method
_, param_dict = parse_user_args(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/dataset/core/validator_helpers.py", line 472, in parse_user_args
ba = sig.bind(method, *args, **kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3186, in bind
return self._bind(args, kwargs)
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/inspect.py", line 3101, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'dataset_files'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 271, in <module>
main(config_)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 44, in wrapper
raise exc
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
result = run_func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/run_mindformer.py", line 45, in main
trainer.predict(predict_checkpoint=config.load_checkpoint, input_data=config.input_data,
File "/home/linweibin/anaconda3/envs/policy/lib/python3.10/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
return func(*args, **kwargs)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/trainer/trainer.py", line 722, in predict
input_data = build_dataset_loader(self.config.eval_dataset.data_loader)
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/dataset/dataloader/build_dataloader.py", line 52, in build_dataset_loader
return MindFormerRegister.get_instance_from_cfg(
File "/home/linweibin/liujian/project/policy-model/mindformers/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
TypeError: MindDataset: missing a required argument: 'dataset_files'
```
dataset_files是干嘛的，推理的话要数据集嘛。
```
(policy) linweibin@user:~/liujian/project/policy-model/mindformers$ bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict"
No parameter is entered. Notice that the program will run on default 8 cards.
scripts/msrun_launcher.sh: line 119: ulimit: max user processes: cannot modify limit: Operation not permitted
Running Command: msrun --worker_num=8 --local_worker_num=8 --master_port=8118 --log_dir=output/msrun_log --join=False --cluster_time_out=600 run_mindformer.py --config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml --run_mode predict
Please check log files in output/msrun_log
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.870.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:0, log file:output/msrun_log/worker_0.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.878.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:1, log file:output/msrun_log/worker_1.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.884.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:2, log file:output/msrun_log/worker_2.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.889.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:3, log file:output/msrun_log/worker_3.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.894.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:4, log file:output/msrun_log/worker_4.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.898.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:5, log file:output/msrun_log/worker_5.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.903.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:6, log file:output/msrun_log/worker_6.log. Environment variable [RANK_ID] is exported.
[WARNING] ME(1512124:281473669746720,MainProcess):2024-09-02-16:32:27.908.000 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:7, log file:output/msrun_log/worker_7.log. Environment variable [RANK_ID] is exported.
(policy) linweibin@user:~/liujian/project/policy-model/mindformers$ bash scripts/msrun_launcher.sh "run_mindformer.py \
--config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml \
--run_mode predict" 1
scripts/msrun_launcher.sh: line 119: ulimit: max user processes: cannot modify limit: Operation not permitted
Running Command: msrun --worker_num=1 --local_worker_num=1 --master_port=8118 --log_dir=output/msrun_log --join=False --cluster_time_out=600 run_mindformer.py --config /home/linweibin/liujian/project/policy-model/mindformers/model/predict_llama3_8b_800T_A2_64G.yaml --run_mode predict
Please check log files in output/msrun_log
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:518: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/linweibin/.local/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
[WARNING] ME(1520522:281473071022112,MainProcess):2024-09-02-16:44:04.220.00 [mindspore/parallel/cluster/process_entity/_api.py:223] Start worker process with rank id:0, log file:output/msrun_log/worker_0.log. Environment variable [RANK_ID] is exported.
```