在使用晟腾的pytorch在910B上做resnet50的训练中。在启动optimizer.step()过程中报如下错误:
File "/usr/local/lib/python3.10/site-packages/torch/optim/sgd.py", line 293, in _multi_tensor_sgd
device_grads = torch._foreach_add(device_grads, device_params, alpha=weight_decay)
RuntimeError: call aclnnForeachAddList failed, detail:EZ1001: [PID: 1165457] 2024-11-27-15:18:33.749.618 Io input dtype or format not support, get io input info is x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(UNDEFINED, ND) out(DT_FLOAT, ND) but support list is:
TraceBack (most recent call last):
DataTypeToSerialString: datatype not support 381622840[FUNC:DataTypeToSerialString][FILE:type_utils.cc][LINE:288]
Io input dtype or format not support, get io input info is x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(UNDEFINED, ND) out(DT_FLOAT, ND) but support list is:
SupportInfo[0] x1(DT_FLOAT16, ND) x2(DT_FLOAT16, ND) alpha(DT_FLOAT, ND) out(DT_FLOAT16, ND)
SupportInfo[1] x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(DT_FLOAT, ND) out(DT_FLOAT, ND)
SupportInfo[2] x1(DT_INT32, ND) x2(DT_INT32, ND) alpha(DT_INT64, ND) out(DT_INT32, ND)
SupportInfo[3] x1(DT_BFLOAT16, ND) x2(DT_BFLOAT16, ND) alpha(DT_FLOAT, ND) out(DT_BFLOAT16, ND)
Check NnopbaseExecutorTilingAndUpdateBinInfo(executor) failed
在使用晟腾的pytorch在910B上做resnet50的训练中。在启动optimizer.step()过程中报如下错误:
File "/usr/local/lib/python3.10/site-packages/torch/optim/sgd.py", line 293, in _multi_tensor_sgd
device_grads = torch._foreach_add(device_grads, device_params, alpha=weight_decay)
RuntimeError: call aclnnForeachAddList failed, detail:EZ1001: [PID: 1165457] 2024-11-27-15:18:33.749.618 Io input dtype or format not support, get io input info is x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(UNDEFINED, ND) out(DT_FLOAT, ND) but support list is:
TraceBack (most recent call last):
DataTypeToSerialString: datatype not support 381622840[FUNC:DataTypeToSerialString][FILE:type_utils.cc][LINE:288]
Io input dtype or format not support, get io input info is x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(UNDEFINED, ND) out(DT_FLOAT, ND) but support list is:
SupportInfo[0] x1(DT_FLOAT16, ND) x2(DT_FLOAT16, ND) alpha(DT_FLOAT, ND) out(DT_FLOAT16, ND)
SupportInfo[1] x1(DT_FLOAT, ND) x2(DT_FLOAT, ND) alpha(DT_FLOAT, ND) out(DT_FLOAT, ND)
SupportInfo[2] x1(DT_INT32, ND) x2(DT_INT32, ND) alpha(DT_INT64, ND) out(DT_INT32, ND)
SupportInfo[3] x1(DT_BFLOAT16, ND) x2(DT_BFLOAT16, ND) alpha(DT_FLOAT, ND) out(DT_BFLOAT16, ND)
Check NnopbaseExecutorTilingAndUpdateBinInfo(executor) failed