昇腾论坛

1、图像分类训练失败，一直卡在导入数据这一步，准备的数据集没有webg格式，报错如红色提示。

2、准备的数据标本20个，每个标本10张图片

/etc/bashrc: line 101: /usr/local/Ascend/ascend-toolkit/set_env.sh: No such file or directory

/home/ma-user/.bashrc: line 6: /usr/local/Ascend/ascend-toolkit/set_env.sh: No such file or directory

INFO:root:Using MoXing-v2.1.1.99185273-99185273

INFO:root:Using OBS-Python-SDK-3.20.9.1

2024-07-11 09:55:54,358 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:<module>:16

INFO:utils.logger:ROOT: /home/ma-user/modelarts/user-job-dir/train

2024-07-11 09:55:54,362 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] train.py:<module>:22

INFO:utils.logger:the current path is: /home/ma-user/modelarts/user-job-dir/train

2024-07-11 09:55:54,365 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] train.py:main:41

INFO:utils.logger:Customer arguments: Namespace(checkpoint_url='1', dataset_url='/test-manstro/datasets/ivp-dataset-7fpl/out/ivp-dataset-2m4j-8bC5DX3vOzO4062t723/annotation/v1/v1.manifest:8bC5DX3vOzO4062t723:o17xRX7ecSN1437Cger', epochs=30, is_hpo=False, status_obs_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', status_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', train_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/model_outputs_url/')

2024-07-11 09:55:54,365 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] train.py:main:44

INFO:utils.logger:ROOT: /home/ma-user/modelarts/user-job-dir/train

{

"status": "completed",

"group_count": "1",

"group_list": [

{

"group_name": "worker",

"device_count": "1",

"instance_count": "1",

"instance_list": [

{

"pod_name": "ma-job-8b879336-b6ce-4501-b2f5-9038082bcf48-worker-0",

"server_id": "100.129.152.171",

"devices": [

{

"device_id": "7",

"device_ip": "192.1.141.31"

}

]

}

]

}

]

}

{

"status": "completed",

"version": "1.0",

"server_count": "1",

"server_list": [

{

"server_id": "100.129.152.171",

"device": [

{

"device_id": "7",

"device_ip": "192.1.141.31",

"rank_id": "0"

}

]

}

]

}

{

"server_id": "100.129.152.171",

"device": [

{

"device_id": "7",

"device_ip": "192.1.141.31",

"rank_id": "0"

}

]

}

entrance in acquire job id ...

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/pandas/compat/numpy/__init__.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

_nlv = LooseVersion(_np_version)

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/pandas/compat/numpy/function.py:125: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(_np_version) >= LooseVersion("1.17.0"):

job id generate success, the job id is: 8b879336-b6ce-4501-b2f5-9038082bcf48

2024-07-11 09:57:27,960 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] train.py:main:86

INFO:utils.logger:Start downloading data from obs

2024-07-11 09:57:28,108 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] pre_processing.py:producer:74

INFO:utils.logger:424 samples will be processed

the current path is: /home/ma-user

2024-07-11 09:57:28,319 INFO [Process:Process-37-1154][Thread:MainThread-281473707401232] pre_processing.py:consumer:102

INFO:utils.logger:200 samples were processed

2024-07-11 09:57:28,361 INFO [Process:Process-18-1135][Thread:MainThread-281473707401232] pre_processing.py:consumer:102

INFO:utils.logger:300 samples were processed

2024-07-11 09:57:28,604 INFO [Process:Process-44-1161][Thread:MainThread-281473707401232] pre_processing.py:consumer:102

INFO:utils.logger:100 samples were processed

2024-07-11 09:57:28,673 INFO [Process:Process-83-1200][Thread:MainThread-281473707401232] pre_processing.py:consumer:102

INFO:utils.logger:400 samples were processed

2024-07-11 09:57:30,487 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] pre_processing.py:run:121

INFO:utils.logger:all samples processed

2024-07-11 09:57:30,489 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] train.py:main:100

INFO:utils.logger:End downloading data

2024-07-11 09:57:30,504 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:run_with_determind_hp:247

INFO:utils.logger:run with determind hp accepted arguments: Namespace(checkpoint_url='1', dataset_url='/test-manstro/datasets/ivp-dataset-7fpl/out/ivp-dataset-2m4j-8bC5DX3vOzO4062t723/annotation/v1/v1.manifest:8bC5DX3vOzO4062t723:o17xRX7ecSN1437Cger', epochs=30, is_hpo=False, progress_bar=10, status_obs_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', status_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', train_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/model_outputs_url/')

2024-07-11 09:57:30,504 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:_check_available_devices:22

INFO:utils.logger:Check available devices ~

2024-07-11 09:57:30,571 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:_check_available_devices:34

INFO:utils.logger:Available devices [0]

2024-07-11 09:57:30,572 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:run_with_determind_hp:255

INFO:utils.logger:Final stage: # available devices {1}, # todo tasks {1}

2024-07-11 09:57:30,583 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:run_with_determind_hp:298

INFO:utils.logger:HPO {final_result} Wait for all task finished: {1} tasks in doing state

{

"status": "completed",

"group_count": "1",

"group_list": [

{

"group_name": "worker",

"device_count": "1",

"instance_count": "1",

"instance_list": [

{

"pod_name": "ma-job-8b879336-b6ce-4501-b2f5-9038082bcf48-worker-0",

"server_id": "100.129.152.171",

"devices": [

{

"device_id": "7",

"device_ip": "192.1.141.31"

}

]

}

]

}

]

}

{

"status": "completed",

"version": "1.0",

"server_count": "1",

"server_list": [

{

"server_id": "100.129.152.171",

"device": [

{

"device_id": "7",

"device_ip": "192.1.141.31",

"rank_id": "0"

}

]

}

]

}

{

"server_id": "100.129.152.171",

"device": [

{

"device_id": "7",

"device_ip": "192.1.141.31",

"rank_id": "0"

}

]

}

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/pandas/compat/numpy/__init__.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

_nlv = LooseVersion(_np_version)

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)

/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/pandas/compat/numpy/function.py:125: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(_np_version) >= LooseVersion("1.17.0"):

2024-07-11 09:57:37,810 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] dispatcher.py:<module>:16

INFO:utils.logger:ROOT: /home/ma-user/modelarts/user-job-dir/train

2024-07-11 09:57:37,971 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] build_model.py:<module>:22

INFO:utils.logger:CUR ROOT: /home/ma-user/modelarts/user-job-dir/train/train_model

the current path is: /home/ma-user/modelarts/workspace/device0

2024-07-11 09:57:37,977 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] train_exec.py:<module>:171

INFO:utils.logger:Stage: final_result Accepted arguments: Namespace(batch_size=64, checkpoint_url='1', color_jitter=1, data_split=None, dataset_url='/home/ma-user/modelarts/user-job-dir/train/dataset', db_file='/home/ma-user/modelarts/user-job-dir/train/hpo_07-11-2024_09-57-30.db', epochs=30, flip=1, hpo_epochs=10, label_smoothing=0.1, layer_decay=0.75, lr=0.0005, progress_bar=10, rrcrop=0, stage='final_result', status_obs_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', status_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/train_status_url/', taskid=1, train_url='obs://test-manstro/datasets/ivp-dataset-7fpl/model_out/4ac61027-50a7-4204-ab51-864cfa63ab53/model_outputs_url/', update_train_process=True, weight_decay=0.05)

2024-07-11 09:57:54,007 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] train_exec.py:main:91

INFO:utils.logger:Start data processing...

use ds.GeneratorDataset, self.image_id:None

2024-07-11 09:57:54,029 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] data_processing.py:run:63

INFO:utils.logger:There are 20 classes

2024-07-11 09:57:54,029 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] data_processing.py:run:64

INFO:utils.logger:Train sample size 333

2024-07-11 09:57:54,029 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] data_processing.py:run:65

INFO:utils.logger:Test sample size 91

2024-07-11 09:57:54,030 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] train_exec.py:main:97

INFO:utils.logger:End data processing...

2024-07-11 09:57:54,030 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] train_exec.py:main:99

INFO:utils.logger:Start model training...

2024-07-11 09:57:54,032 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] build_model.py:__init__:66

INFO:utils.logger:Found Pangu L1 model in local path

2024-07-11 09:57:58,384 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] build_model.py:run:75

INFO:utils.logger:Loading pretrained model...

2024-07-11 09:58:02,337 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] build_model.py:run:82

INFO:utils.logger:Finished loading...

2024-07-11 09:58:04,755 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] build_model.py:run:139

INFO:utils.logger:Start training...

Line of code : 2400

File : mindspore/ccsrc/minddata/dataset/kernels/image/image_utils.cc

Traceback (most recent call last):

File "/home/ma-user/modelarts/user-job-dir/train/train_exec.py", line 174, in <module>

main(parsed_args)

File "/home/ma-user/modelarts/user-job-dir/train/train_exec.py", line 100, in main

model = BuildModel(data_processing, args, training_progress_bar).run()

File "/home/ma-user/modelarts/user-job-dir/train/train_model/build_model.py", line 148, in run

dataset_sink_mode=False)

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/train/model.py", line 1073, in train

initial_epoch=initial_epoch)

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/train/model.py", line 114, in wrapper

func(self, *args, **kwargs)

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/train/model.py", line 617, in _train

self._train_process(epoch, train_dataset, list_callback, cb_params, initial_epoch, valid_infos)

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/train/model.py", line 906, in _train_process

for next_element in dataset_helper:

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/train/dataset_helper.py", line 638, in __next__

data = self.iter.__next__()

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 152, in __next__

data = self._get_next()

File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 301, in _get_next

return [self._transform_md_to_output(t) for t in self._iterator.GetNextAsList()]

RuntimeError: Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.

------------------------------------------------------------------

- Dataset Pipeline Error Message:

------------------------------------------------------------------

- C++ Call Stack: (For framework developers)

------------------------------------------------------------------

mindspore/ccsrc/minddata/dataset/kernels/image/image_utils.cc(2400).

2024-07-11 09:58:04,999 INFO [Process:MainProcess-1289][Thread:MainThread-281473124929552] train_exec.py:<module>:178

INFO:utils.logger:Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.

------------------------------------------------------------------

- Dataset Pipeline Error Message:

------------------------------------------------------------------

- C++ Call Stack: (For framework developers)

------------------------------------------------------------------

mindspore/ccsrc/minddata/dataset/kernels/image/image_utils.cc(2400).

2024-07-11 09:58:30,643 INFO [Process:MainProcess-260][Thread:MainThread-281473707401232] dispatcher.py:run_with_determind_hp:305

INFO:utils.logger:[(1, "error:Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'. \n\n------------------------------------------------------------------\n- Dataset Pipeline Error Message: \n------------------------------------------------------------------\n[ERROR] map operation: [Decode] failed. Unsupported image type [webp], but dump the image failed. Error info: WriteFile: The type of the elements of data should be UINT8, but got int8.\n\n------------------------------------------------------------------\n- C++ Call Stack: (For framework developers) \n------------------------------------------------------------------\nmindspore/ccsrc/minddata/dataset/kernels/image/image_utils.cc(2400).\n\n\n", 0, 0.0005, 0.05, 0.75, 0.1, 0, 1, 1, 0.0)]

the training process is: {'processed': 10, 'total': 100, 'msg': '', 'info': {}}

Traceback (most recent call last):

File "/home/ma-user/modelarts/user-job-dir/train/train.py", line 156, in <module>

main()

File "/home/ma-user/modelarts/user-job-dir/train/train.py", line 152, in main

run_with_determind_hp(args, db_file, 'final_result')

File "/home/ma-user/modelarts/user-job-dir/train/dispatcher.py", line 306, in run_with_determind_hp

raise RuntimeError(tasks_error[0][1])

RuntimeError: error:Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.

------------------------------------------------------------------

- Dataset Pipeline Error Message:

------------------------------------------------------------------

- C++ Call Stack: (For framework developers)

------------------------------------------------------------------

mindspore/ccsrc/minddata/dataset/kernels/image/image_utils.cc(2400).

The parse job starts. Please wait.

The parse job is complete.

The diag job starts. Please wait.

+------------------------------------------------------------------------------------------------------------------------------------+

| Ascend Fault-Diag Report |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| 版本信息 | 5.0.0 | |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| 根因节点分析 | 类型 | 描述 |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| | 说明 | 未诊断出根因节点，根因设备分析将尝试检测全部设备 |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| | 根因节点 | ['Unknown Device'] |

| | 现象描述 | 此训练任务可能是单卡训练任务，或者是HCCL初始化失败。无法获取有效的卡信息。请直接排查任务所使用的对应卡。 |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| 根因设备分析 | 类型 | 描述 |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

| | 状态码 | 21029 |

| | 根因设备 | ['worker-0'] |

| | 故障分类 | 类别:Software 组件:AI Framework 模块:Dataset |

| | 故障名称 | 数据集加载与处理故障 |

| | 故障描述 | 识别到当前报错可能为数据集加载与处理故障。 |

| | 建议方案 | 1. 请参考MindSpore数据集故障处理指导进行问题排查或者在MindSpore社区求助； |

| | | 2. 论坛-数据处理与加载： |

| | | https://bbs.huaweicloud.com/forum/forum-1076-2476-1.html |

| | | 3. Dataset FAQ： |

| | | https://www.mindspore.cn/docs/zh-CN/r1.8/faq/data_processing.html |

| | | 4. Dataset 错误分析： |

| | | https://www.mindspore.cn/tutorials/experts/zh-CN/r1.8/debug/error_analyze.html |

+--------------+----------+----------------------------------------------------------------------------------------------------------+

The diag job is complete.

关于昇腾

新闻与活动

交流与资讯

支持与服务

开源社区