训练时出现/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:203 NPU error, error code is 500002
收藏回复举报
训练时出现/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:203 NPU error, error code is 500002
新人帖
发表于2024-01-11 17:48:48
0 查看

反向传播时 loss.backward()出现以下错误,求解决

Traceback (most recent call last): 

  File "/root/project/trainbertclassify.py", line 168, in <module> 

    loss.backward() 

  File "/root/anaconda3/envs/test/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward 

    torch.autograd.backward( 

  File "/root/anaconda3/envs/test/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward 

    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass 

RuntimeError: InnerRun:/home/test2/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:203 NPU error, error code is 500002 

[Error]: A GE error occurs in the system. 

        Rectify the fault based on the error information in the ascend log. 

EZ3003: No supported Ops kernel and engine are found for [EmbeddingDenseGrad], optype [EmbeddingDenseGrad]. 

        Possible Cause: The operator is not supported by the system. Therefore, no hit is found in any operator information library. 

        Solution: 1. Check that the OPP component is installed properly. 2. Submit an issue to request for the support of this operator type. 

        TraceBack (most recent call last): 

        build graph failed, graph id:76, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615] 

        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] 

        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

本帖最后由 匿名用户2024/01/11 19:52:30 编辑

我要发帖子