stub_tensor 获取shape异常问题
收藏回复举报
stub_tensor 获取shape异常问题
t('forum.solved') 已解决
发表于2024-03-18 19:39:20
0 查看

环境

目前在gpu环境中开启梯度训练出现该问题,而仅推理和cpu上不会出现

//开启梯度训练
(loss), grads = grad_fn(mini_batch)

mindspore 2.2.1 显卡 RTX3090 操作系统 Ubuntu20.04 cuda:12.0,下载的是11.6版本的

问题

随着训练迭代进行,在第2,3个epho上会在以下非梯度计算位置出现报错,但代码确实没问题(cpu和推理)

//报错位置
---> 84 deg_inv[deg_inv == float('inf')] = 0 
//报错原因
   2697 def __call__(self, data, index, value):
-> 2698     return Tensor_.setitem_index_info(data, index, value, self.is_ascend)
RuntimeError: Output device address's size 1 is not equal output_size_list's size 2
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/runtime/pynative/run_op_helper.cc:644 UpdateOutputDeviceInfo

当注释掉引发错误的一整个代码段后,又出现了同样的报错UpdateOutputDeviceInfo,这里显示是获取.shape错误,但这里不使用shape后又会出现同样的UpdateOutputDeviceInfo报错,但报错位置不同

//报错位置
File "/root/mbe-mindspore/mindspore_src/utils/opsi.py", line 165, in tile_along_beam
for i, d in enumerate(v.shape):
File "/root/miniconda3/lib/python3.8/site-packages/mindspore/common/_stub_tensor.py", line 85, in shape
self.stub_shape = self.stub.get_shape()
//报错信息
RuntimeError: Output device address's size 1 is not equal output_size_list's size 2
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/runtime/pynative/run_op_helper.cc:644 UpdateOutputDeviceInfo

我要发帖子