(mindspore) HwHiAiUser@orangepiaipro-20t:~/Downloads/git_imag$ python main.py -a resnet50 -b 32 --gpu 0 --dummy
/home/HwHiAiUser/anaconda3/envs/mindspore/lib/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:288: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/home/HwHiAiUser/anaconda3/envs/mindspore/lib/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:247: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/home/HwHiAiUser/Downloads/git_imag/main.py:102: UserWarning: You have chosen a specific GPU. This will completely disable data parallelism.
warnings.warn('You have chosen a specific GPU. This will completely '
/home/HwHiAiUser/Downloads/git_imag/main.py:113: UserWarning: nccl backend >=2.5 requires GPU count>1, see https://github.com/NVIDIA/nccl/issues/103 perhaps use 'gloo'
warnings.warn("nccl backend >=2.5 requires GPU count>1, see https://github.com/NVIDIA/nccl/issues/103 perhaps use 'gloo'")
Use GPU: 0 for training
=> creating model 'resnet50'
.=> Dummy data is used!
/home/HwHiAiUser/anaconda3/envs/mindspore/lib/python3.9/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 3, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
.[W compiler_depend.ts:387] Warning: EZ3002: [PID: 41553] 2025-04-08-16:31:03.481.704 Optype [MaxPoolWithArgmaxV1] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type MaxPoolWithArgmaxV1 is not found in this op store.[tbe-custom]:op type MaxPoolWithArgmaxV1 is not found in this op store.[Dynamic shape check]: data type DT_FLOAT of input [x] is not supported. All supported data type and format of tensor input0.x is: Data Type: {DT_FLOAT16}Format:{NC1HWC0}[Static shape check]:data type DT_FLOAT of input [x] is not supported. All supported data type and format of tensor input0.x is: Data Type: {DT_FLOAT16}Format:{NC1HWC0}.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
No supported Ops kernel and engine are found for [MaxPoolWithArgmaxV132], optype [MaxPoolWithArgmaxV1].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library[Static shape check]:The format and dtype is not precisely equivalent to format and dtype in op information library.
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_48], optype [TransData].
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:31, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
(function ExecFunc)
Traceback (most recent call last):
File "/home/HwHiAiUser/Downloads/git_imag/main.py", line 517, in <module>
main()
File "/home/HwHiAiUser/Downloads/git_imag/main.py", line 126, in main
main_worker(args.gpu, ngpus_per_node, args)
File "/home/HwHiAiUser/Downloads/git_imag/main.py", line 284, in main_worker
train(train_loader, model, criterion, optimizer, epoch, device, args)
File "/home/HwHiAiUser/Downloads/git_imag/main.py", line 336, in train
losses.update(loss.item(), images.size(0))
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is MaxPoolWithArgmaxV1.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-08-16:31:03 (PID:41553, Device:0, RankID:-1) ERR00100 PTA call acl api failed
我按照文档中的方法进行 PyTorch 网络迁移出现问题:
环境说明:
使用 PyTorch GPU2Ascend 工具迁移
main.py脚本;CANN 驱动和
torch_npu插件已安装;使用 dummy 数据进行验证训练;
当前运行环境为 香橙派aipro(20t),芯片为310B,单卡 ID=0;
运行命令:
python main.py -a resnet50 -b 32 --gpu 0 --dummy已完成操作:
成功导入
torch_npu;插入迁移代码
import torch_npu与from torch_npu.contrib import transfer_to_npu;能成功启动训练任务,模型能够被创建;
dummy 数据也可以被加载;
也尝试过使用
resnet18替代模型;报错信息(resnet50 和 resnet18 均报错):
尝试解决方法:
已切换为
resnet18模型;已设置
--gpu 0;运行环境变量已执行
source set_env.sh;但问题依旧存在。
烦请协助确认是什么问题:感谢支持!