[W AmpForeachNonFiniteCheckAndUnscaleKernelNpu.cpp:46] Warning: Non finite check and unscale on NPU device! (function operator())
Traceback (most recent call last):
File "/home/ma-user/work/test.py", line 68, in <module>
scaler.step(optimizer) # 更新参数(自动unscaling)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch_npu/npu/amp/grad_scaler.py", line 402, in step
retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch_npu/npu/amp/grad_scaler.py", line 345, in _maybe_opt_step
retval = optimizer.step(*args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 280, in wrapper
out = func(*args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 33, in _use_grad
ret = func(self, *args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 76, in step
sgd(params_with_grad,
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 222, in sgd
func(params,
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 325, in _multi_tensor_sgd
torch._foreach_add_(device_params, device_grads, alpha=-lr)
RuntimeError: call aclnnForeachAddList failed, detail:EZ1001: Get path and read config failed.
TraceBack (most recent call last):
Check NnopbaseCollecterWork(binCollecter.get()) failed
Assert ((NnopbaseInit()) == 0) failed
Check NnopbaseCreateExecutorSpace(&executorSpace) failed
Traceback (most recent call last):
File "/home/ma-user/work/test.py", line 68, in <module>
scaler.step(optimizer) # 更新参数(自动unscaling)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch_npu/npu/amp/grad_scaler.py", line 402, in step
retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch_npu/npu/amp/grad_scaler.py", line 345, in _maybe_opt_step
retval = optimizer.step(*args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 280, in wrapper
out = func(*args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 33, in _use_grad
ret = func(self, *args, **kwargs)
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 76, in step
sgd(params_with_grad,
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 222, in sgd
func(params,
File "/home/ma-user/anaconda3/lib/python3.9/site-packages/torch/optim/sgd.py", line 325, in _multi_tensor_sgd
torch._foreach_add_(device_params, device_grads, alpha=-lr)
RuntimeError: call aclnnForeachAddList failed, detail:EZ1001: Get path and read config failed.
TraceBack (most recent call last):
Check NnopbaseCollecterWork(binCollecter.get()) failed
Assert ((NnopbaseInit()) == 0) failed
Check NnopbaseCreateExecutorSpace(&executorSpace) failed