运行效果如下，用的是Imagenet数据集，就搞了100个类型的图片分类，不知道为什么包下面这个错误：
{}
{'autoaugment': 1,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': -1,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
Please check the above information for the configurations
standalone training
########################################
{'auto_tune': 0,
'autoaugment': 1,
'aux_factor': 0.4,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_name': 'imagenet',
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'device_id': 0,
'device_num': 1,
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_engine': 'imagenet',
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': 0,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'local_rank': 0,
'logger': <LOGGER mindspore-benchmark (NOTSET)>,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'poly_power': 2,
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'seed': 1,
'split_point': 0.4,
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
########################################
get vit_config_path
split_point=0.4
============================================================
{'auto_tune': 0,
'autoaugment': 1,
'aux_factor': 0.4,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_name': 'imagenet',
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'device_id': 0,
'device_num': 1,
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_engine': 'imagenet',
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': 0,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'local_rank': 0,
'logger': <LOGGER mindspore-benchmark (NOTSET)>,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'poly_power': 2,
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'seed': 1,
'split_point': 0.4,
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
============================================================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
100
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:12.107.309 [mindspore/dataset/engine/datasets.py:2603] Repeat is located before batch, data from two epochs can be batched together.
eval batch per step: 4
eval dataset num_padded: 0
eval dataset size: 5050
eval, resize:256
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:12.130.515 [mindspore/dataset/engine/datasets.py:2603] Repeat is located before batch, data from two epochs can be batched together.
2024-06-17 11:51:14:INFO:Args:
2024-06-17 11:51:14:INFO:--> enable_modelarts: 0
2024-06-17 11:51:14:INFO:--> dataset_path: /data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train
2024-06-17 11:51:14:INFO:--> train_image_size: 224
2024-06-17 11:51:14:INFO:--> interpolation: BILINEAR
2024-06-17 11:51:14:INFO:--> crop_min: 0.05
2024-06-17 11:51:14:INFO:--> batch_size: 8
2024-06-17 11:51:14:INFO:--> train_num_workers: 1
2024-06-17 11:51:14:INFO:--> eval_path: /data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val
2024-06-17 11:51:14:INFO:--> eval_image_size: 224
2024-06-17 11:51:14:INFO:--> eval_batch_size: 4
2024-06-17 11:51:14:INFO:--> eval_interval: 1
2024-06-17 11:51:14:INFO:--> eval_offset: 0
2024-06-17 11:51:14:INFO:--> eval_num_workers: 2
2024-06-17 11:51:14:INFO:--> backbone: vit_base_patch32
2024-06-17 11:51:14:INFO:--> class_num: 100
2024-06-17 11:51:14:INFO:--> vit_config_path: src.vit.VitConfig
2024-06-17 11:51:14:INFO:--> pretrained:
2024-06-17 11:51:14:INFO:--> lr_decay_mode: cosine
2024-06-17 11:51:14:INFO:--> lr_init: 0.001
2024-06-17 11:51:14:INFO:--> lr_max: 0.00044375
2024-06-17 11:51:14:INFO:--> lr_min: 1e-08
2024-06-17 11:51:14:INFO:--> max_epoch: 300
2024-06-17 11:51:14:INFO:--> warmup_epochs: 0
2024-06-17 11:51:14:INFO:--> opt: adamw
2024-06-17 11:51:14:INFO:--> beta1: 0.9
2024-06-17 11:51:14:INFO:--> beta2: 0.999
2024-06-17 11:51:14:INFO:--> weight_decay: 0.05
2024-06-17 11:51:14:INFO:--> no_weight_decay_filter: beta,bias
2024-06-17 11:51:14:INFO:--> gc_flag: 0
2024-06-17 11:51:14:INFO:--> loss_scale: 1024
2024-06-17 11:51:14:INFO:--> use_label_smooth: 1
2024-06-17 11:51:14:INFO:--> label_smooth_factor: 0.1
2024-06-17 11:51:14:INFO:--> mixup: 0.2
2024-06-17 11:51:14:INFO:--> autoaugment: 1
2024-06-17 11:51:14:INFO:--> loss_name: ce_smooth_mixup
2024-06-17 11:51:14:INFO:--> save_checkpoint: 1
2024-06-17 11:51:14:INFO:--> save_checkpoint_epochs: 8
2024-06-17 11:51:14:INFO:--> keep_checkpoint_max: 3
2024-06-17 11:51:14:INFO:--> save_checkpoint_path: ./outputs
2024-06-17 11:51:14:INFO:--> open_profiler: 0
2024-06-17 11:51:14:INFO:--> config_path: vit_patch32_imagenet2012_config_standalone.yml
2024-06-17 11:51:14:INFO:--> eval_engine:
2024-06-17 11:51:14:INFO:--> split_point: 0.4
2024-06-17 11:51:14:INFO:--> poly_power: 2
2024-06-17 11:51:14:INFO:--> aux_factor: 0.4
2024-06-17 11:51:14:INFO:--> seed: 1
2024-06-17 11:51:14:INFO:--> auto_tune: 0
2024-06-17 11:51:14:INFO:--> device_id: 0
2024-06-17 11:51:14:INFO:--> local_rank: 0
2024-06-17 11:51:14:INFO:--> device_num: 1
2024-06-17 11:51:14:INFO:--> dataset_name: imagenet
2024-06-17 11:51:14:INFO:--> logger: <LOGGER mindspore-benchmark (NOTSET)>
2024-06-17 11:51:14:INFO:--> steps_per_epoch: 6312
2024-06-17 11:51:14:INFO:
2024-06-17 11:51:14:INFO:compile time used=0.00s
我的sink_size是6312
我的callbacks是[<src.callback.StateMonitor object at 0xfffdbf3d9bd0>, <mindspore.train.callback._checkpoint.ModelCheckpoint object at 0xfffdbf3d9f50>]
我的epoch_size是300
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:14.243.862 [mindspore/train/model.py:1108] For StateMonitor callback, {'step_end', 'epoch_end', 'epoch_begin'} methods may not be supported in later version, Use methods prefixed with 'on_train' or 'on_eval' instead when using customized callbacks.
[ERROR] MD(77569,fffc7b7ef200,python3):2024-06-17-11:51:15.159.685 [mindspore/ccsrc/minddata/dataset/util/task_manager.cc:227] InterruptMaster] MindSpore dataset is terminated with err msg: Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'. map operation: [OneHot] failed. OneHot: index values should not bigger than num classes: 100, but got: 100
Line of code : 49
File : mindspore/ccsrc/minddata/dataset/kernels/data/data_utils.cc
点赞
收藏
回复
分享
举报
已解决
运行效果如下，用的是Imagenet数据集，就搞了100个类型的图片分类，不知道为什么包下面这个错误：
{}
{'autoaugment': 1,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': -1,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
Please check the above information for the configurations
standalone training
########################################
{'auto_tune': 0,
'autoaugment': 1,
'aux_factor': 0.4,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_name': 'imagenet',
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'device_id': 0,
'device_num': 1,
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_engine': 'imagenet',
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': 0,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'local_rank': 0,
'logger': <LOGGER mindspore-benchmark (NOTSET)>,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'poly_power': 2,
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'seed': 1,
'split_point': 0.4,
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
########################################
get vit_config_path
split_point=0.4
============================================================
{'auto_tune': 0,
'autoaugment': 1,
'aux_factor': 0.4,
'backbone': 'vit_base_patch32',
'batch_size': 8,
'beta1': 0.9,
'beta2': 0.999,
'class_num': 100,
'config_path': 'vit_patch32_imagenet2012_config_standalone.yml',
'crop_min': 0.05,
'dataset_name': 'imagenet',
'dataset_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train',
'device_id': 0,
'device_num': 1,
'enable_modelarts': 0,
'eval_batch_size': 4,
'eval_engine': 'imagenet',
'eval_image_size': 224,
'eval_interval': 1,
'eval_num_workers': 2,
'eval_offset': 0,
'eval_path': '/data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val',
'gc_flag': 0,
'interpolation': 'BILINEAR',
'keep_checkpoint_max': 3,
'label_smooth_factor': 0.1,
'local_rank': 0,
'logger': <LOGGER mindspore-benchmark (NOTSET)>,
'loss_name': 'ce_smooth_mixup',
'loss_scale': 1024,
'lr_decay_mode': 'cosine',
'lr_init': 0.001,
'lr_max': 0.00044375,
'lr_min': 1e-08,
'max_epoch': 300,
'mixup': 0.2,
'no_weight_decay_filter': 'beta,bias',
'open_profiler': 0,
'opt': 'adamw',
'poly_power': 2,
'pretrained': '',
'save_checkpoint': 1,
'save_checkpoint_epochs': 8,
'save_checkpoint_path': './outputs',
'seed': 1,
'split_point': 0.4,
'train_image_size': 224,
'train_num_workers': 1,
'use_label_smooth': 1,
'vit_config_path': 'src.vit.VitConfig',
'warmup_epochs': 0,
'weight_decay': 0.05}
============================================================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
100
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:12.107.309 [mindspore/dataset/engine/datasets.py:2603] Repeat is located before batch, data from two epochs can be batched together.
eval batch per step: 4
eval dataset num_padded: 0
eval dataset size: 5050
eval, resize:256
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:12.130.515 [mindspore/dataset/engine/datasets.py:2603] Repeat is located before batch, data from two epochs can be batched together.
2024-06-17 11:51:14:INFO:Args:
2024-06-17 11:51:14:INFO:--> enable_modelarts: 0
2024-06-17 11:51:14:INFO:--> dataset_path: /data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/train
2024-06-17 11:51:14:INFO:--> train_image_size: 224
2024-06-17 11:51:14:INFO:--> interpolation: BILINEAR
2024-06-17 11:51:14:INFO:--> crop_min: 0.05
2024-06-17 11:51:14:INFO:--> batch_size: 8
2024-06-17 11:51:14:INFO:--> train_num_workers: 1
2024-06-17 11:51:14:INFO:--> eval_path: /data/zx_cv/vit/models-master/VIT/cache/datasets/imagenet_cifa/tiny-new/datasets/val
2024-06-17 11:51:14:INFO:--> eval_image_size: 224
2024-06-17 11:51:14:INFO:--> eval_batch_size: 4
2024-06-17 11:51:14:INFO:--> eval_interval: 1
2024-06-17 11:51:14:INFO:--> eval_offset: 0
2024-06-17 11:51:14:INFO:--> eval_num_workers: 2
2024-06-17 11:51:14:INFO:--> backbone: vit_base_patch32
2024-06-17 11:51:14:INFO:--> class_num: 100
2024-06-17 11:51:14:INFO:--> vit_config_path: src.vit.VitConfig
2024-06-17 11:51:14:INFO:--> pretrained:
2024-06-17 11:51:14:INFO:--> lr_decay_mode: cosine
2024-06-17 11:51:14:INFO:--> lr_init: 0.001
2024-06-17 11:51:14:INFO:--> lr_max: 0.00044375
2024-06-17 11:51:14:INFO:--> lr_min: 1e-08
2024-06-17 11:51:14:INFO:--> max_epoch: 300
2024-06-17 11:51:14:INFO:--> warmup_epochs: 0
2024-06-17 11:51:14:INFO:--> opt: adamw
2024-06-17 11:51:14:INFO:--> beta1: 0.9
2024-06-17 11:51:14:INFO:--> beta2: 0.999
2024-06-17 11:51:14:INFO:--> weight_decay: 0.05
2024-06-17 11:51:14:INFO:--> no_weight_decay_filter: beta,bias
2024-06-17 11:51:14:INFO:--> gc_flag: 0
2024-06-17 11:51:14:INFO:--> loss_scale: 1024
2024-06-17 11:51:14:INFO:--> use_label_smooth: 1
2024-06-17 11:51:14:INFO:--> label_smooth_factor: 0.1
2024-06-17 11:51:14:INFO:--> mixup: 0.2
2024-06-17 11:51:14:INFO:--> autoaugment: 1
2024-06-17 11:51:14:INFO:--> loss_name: ce_smooth_mixup
2024-06-17 11:51:14:INFO:--> save_checkpoint: 1
2024-06-17 11:51:14:INFO:--> save_checkpoint_epochs: 8
2024-06-17 11:51:14:INFO:--> keep_checkpoint_max: 3
2024-06-17 11:51:14:INFO:--> save_checkpoint_path: ./outputs
2024-06-17 11:51:14:INFO:--> open_profiler: 0
2024-06-17 11:51:14:INFO:--> config_path: vit_patch32_imagenet2012_config_standalone.yml
2024-06-17 11:51:14:INFO:--> eval_engine:
2024-06-17 11:51:14:INFO:--> split_point: 0.4
2024-06-17 11:51:14:INFO:--> poly_power: 2
2024-06-17 11:51:14:INFO:--> aux_factor: 0.4
2024-06-17 11:51:14:INFO:--> seed: 1
2024-06-17 11:51:14:INFO:--> auto_tune: 0
2024-06-17 11:51:14:INFO:--> device_id: 0
2024-06-17 11:51:14:INFO:--> local_rank: 0
2024-06-17 11:51:14:INFO:--> device_num: 1
2024-06-17 11:51:14:INFO:--> dataset_name: imagenet
2024-06-17 11:51:14:INFO:--> logger: <LOGGER mindspore-benchmark (NOTSET)>
2024-06-17 11:51:14:INFO:--> steps_per_epoch: 6312
2024-06-17 11:51:14:INFO:
2024-06-17 11:51:14:INFO:compile time used=0.00s
我的sink_size是6312
我的callbacks是[<src.callback.StateMonitor object at 0xfffdbf3d9bd0>, <mindspore.train.callback._checkpoint.ModelCheckpoint object at 0xfffdbf3d9f50>]
我的epoch_size是300
[WARNING] ME(77569:281472473717680,MainProcess):2024-06-17-11:51:14.243.862 [mindspore/train/model.py:1108] For StateMonitor callback, {'step_end', 'epoch_end', 'epoch_begin'} methods may not be supported in later version, Use methods prefixed with 'on_train' or 'on_eval' instead when using customized callbacks.
[ERROR] MD(77569,fffc7b7ef200,python3):2024-06-17-11:51:15.159.685 [mindspore/ccsrc/minddata/dataset/util/task_manager.cc:227] InterruptMaster] MindSpore dataset is terminated with err msg: Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'. map operation: [OneHot] failed. OneHot: index values should not bigger than num classes: 100, but got: 100
Line of code : 49
File : mindspore/ccsrc/minddata/dataset/kernels/data/data_utils.cc