项目源码：https://gitee.com/ascend/ModelZoo-PyTorch
训练启动命令：
# bash ./test/train_full_8p.sh --data_path=./tiny-imagenet-200
device 0日志无错误提示，但卡住了：
Using NVIDIA APEX AMP. Training in mixed precision.
Using NVIDIA APEX DistributedDataParallel.
model is train in distributed
Scheduled epochs: 650
./tiny-imagenet-200/train
./tiny-imagenet-200/val
[W reducer.cpp:401] Warning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.
grad.sizes() = [1000, 1280], strides() = [1280, 1]
bucket_view.sizes() = [1280000], strides() = [1] (function operator())
group num: 2
Reducer buckets have been rebuilt in this iteration.
。。。
Train: 8 [ 23/24 (100%)] Loss: 10576.114258 (10564.8838) Time: 0.178s, 22990.70/s (0.178s, 22984.37/s) LR: 1.599e+00 Data: 0.000 (0.000) FPS: 22984.367 Batch_Size:512.0
Current checkpoints:
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-5.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-6.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-7.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-8.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-0.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-1.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-2.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-3.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-4.pth.tar', 0.0)
device 1、2、5、6均有相同错误，其余卡无错误：
Process ForkServerProcess-2:
Traceback (most recent call last):
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 61, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 58, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 275, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
kind, result = conn.recv()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 91 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
plog日志无错误提示：
device 0日志：
(torch) [root@train plog]# tail plog-328984_20230821105412530.log
[TRACE] GE(328984,python3):2023-08-21-10:54:12.403.523 [status:INIT] [ge_api.cc:200]328984 GEInitializeImpl:GEInitialize start
[TRACE] GE(328984,python3):2023-08-21-10:54:12.580.381 [status:RUNNING] [ge_api.cc:266]328984 GEInitializeImpl:Initializing environment
[TRACE] GE(328984,python3):2023-08-21-10:54:19.921.396 [status:STOP] [ge_api.cc:309]328984 GEInitializeImpl:GEInitialize finished
[TRACE] GE(328984,python3):2023-08-21-10:54:19.922.020 [status:INIT] [ge_api.cc:200]328984 GEInitializeImpl:GEInitialize start
[TRACE] HCCL(328984,python3):2023-08-21-10:54:35.243.598 [status:init] [op_base.cc:158][328984][328984]HcclGetRootInfo success, take time [18013]us
[TRACE] HCCL(328984,python3):2023-08-21-10:54:38.254.859 [status:init] [op_base.cc:267][hccl-328984-0-1692586478-hccl_world_group][0]HcclCommInitRootInfo success,take time [3011083]us, rankNum[8], rank[0],rootInfo identifier[10.42.0.1%cni0_60000_0_1692586475243561], server[10.42.0.1%cni0], device[0]
其余日志均类似：
[TRACE] GE(328999,python3):2023-08-21-10:54:12.432.431 [status:INIT] [ge_api.cc:200]328999 GEInitializeImpl:GEInitialize start
[TRACE] GE(328999,python3):2023-08-21-10:54:12.614.478 [status:RUNNING] [ge_api.cc:266]328999 GEInitializeImpl:Initializing environment
[TRACE] GE(328999,python3):2023-08-21-10:54:19.979.805 [status:STOP] [ge_api.cc:309]328999 GEInitializeImpl:GEInitialize finished
[TRACE] GE(328999,python3):2023-08-21-10:54:19.980.474 [status:INIT] [ge_api.cc:200]328999 GEInitializeImpl:GEInitialize start
[TRACE] HCCL(328999,python3):2023-08-21-10:54:38.194.231 [status:init] [op_base.cc:267][hccl-328999-0-1692586478-hccl_world_group][5]HcclCommInitRootInfo success,take time [2695261]us, rankNum[8], rank[5],rootInfo identifier[10.42.0.1%cni0_60000_0_1692586475243561], server[10.42.0.1%cni0], device[5]
CANN版本：6.3.RC2.alpha003，OS:麒麟V10 SP1
点赞
收藏
回复
分享
举报
已解决
项目源码：https://gitee.com/ascend/ModelZoo-PyTorch
训练启动命令：
# bash ./test/train_full_8p.sh --data_path=./tiny-imagenet-200
device 0日志无错误提示，但卡住了：
Using NVIDIA APEX AMP. Training in mixed precision.
Using NVIDIA APEX DistributedDataParallel.
model is train in distributed
Scheduled epochs: 650
./tiny-imagenet-200/train
./tiny-imagenet-200/val
[W reducer.cpp:401] Warning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.
grad.sizes() = [1000, 1280], strides() = [1280, 1]
bucket_view.sizes() = [1280000], strides() = [1] (function operator())
group num: 2
Reducer buckets have been rebuilt in this iteration.
。。。
Train: 8 [ 23/24 (100%)] Loss: 10576.114258 (10564.8838) Time: 0.178s, 22990.70/s (0.178s, 22984.37/s) LR: 1.599e+00 Data: 0.000 (0.000) FPS: 22984.367 Batch_Size:512.0
Current checkpoints:
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-5.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-6.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-7.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-8.pth.tar', 100.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-0.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-1.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-2.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-3.pth.tar', 0.0)
('./output/train/20230821-105452-mobilenetv3_large_100-224/checkpoint-4.pth.tar', 0.0)
device 1、2、5、6均有相同错误，其余卡无错误：
Process ForkServerProcess-2:
Traceback (most recent call last):
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 61, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 58, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 275, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
kind, result = conn.recv()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
/root/miniconda3/envs/torch/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 91 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
plog日志无错误提示：
device 0日志：
(torch) [root@train plog]# tail plog-328984_20230821105412530.log
[TRACE] GE(328984,python3):2023-08-21-10:54:12.403.523 [status:INIT] [ge_api.cc:200]328984 GEInitializeImpl:GEInitialize start
[TRACE] GE(328984,python3):2023-08-21-10:54:12.580.381 [status:RUNNING] [ge_api.cc:266]328984 GEInitializeImpl:Initializing environment
[TRACE] GE(328984,python3):2023-08-21-10:54:19.921.396 [status:STOP] [ge_api.cc:309]328984 GEInitializeImpl:GEInitialize finished
[TRACE] GE(328984,python3):2023-08-21-10:54:19.922.020 [status:INIT] [ge_api.cc:200]328984 GEInitializeImpl:GEInitialize start
[TRACE] HCCL(328984,python3):2023-08-21-10:54:35.243.598 [status:init] [op_base.cc:158][328984][328984]HcclGetRootInfo success, take time [18013]us
[TRACE] HCCL(328984,python3):2023-08-21-10:54:38.254.859 [status:init] [op_base.cc:267][hccl-328984-0-1692586478-hccl_world_group][0]HcclCommInitRootInfo success,take time [3011083]us, rankNum[8], rank[0],rootInfo identifier[10.42.0.1%cni0_60000_0_1692586475243561], server[10.42.0.1%cni0], device[0]
其余日志均类似：
[TRACE] GE(328999,python3):2023-08-21-10:54:12.432.431 [status:INIT] [ge_api.cc:200]328999 GEInitializeImpl:GEInitialize start
[TRACE] GE(328999,python3):2023-08-21-10:54:12.614.478 [status:RUNNING] [ge_api.cc:266]328999 GEInitializeImpl:Initializing environment
[TRACE] GE(328999,python3):2023-08-21-10:54:19.979.805 [status:STOP] [ge_api.cc:309]328999 GEInitializeImpl:GEInitialize finished
[TRACE] GE(328999,python3):2023-08-21-10:54:19.980.474 [status:INIT] [ge_api.cc:200]328999 GEInitializeImpl:GEInitialize start
[TRACE] HCCL(328999,python3):2023-08-21-10:54:38.194.231 [status:init] [op_base.cc:267][hccl-328999-0-1692586478-hccl_world_group][5]HcclCommInitRootInfo success,take time [2695261]us, rankNum[8], rank[5],rootInfo identifier[10.42.0.1%cni0_60000_0_1692586475243561], server[10.42.0.1%cni0], device[5]
CANN版本：6.3.RC2.alpha003，OS:麒麟V10 SP1