昇腾故障案例详情页

在模型运行时遇到报错“TVM/te/cce error.”

更新时间: 2023/06/06

暂无评分

问题信息

问题来源产品大类关键字
官方模型训练--

问题现象描述

  • 报错截图

  • 报错文本
    Traceback (most recent call last):
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/__init__.py", line 108, in <module>
        __import__('topi.cce')
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/topi/cce/__init__.py", line 20, in <module>
        import te.lang.cce
      File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/__init__.py", line 18, in <module>
        from .te_compute.broadcast_compute import broadcast
      File "/root/archiconda3/envs/***/Lib/python3.7/site-packages/te/lang/cce/te_compute/__init__.py", line 23, in <module>
        from .broadcast_compute import broadcast
      File "/root/archiconda3/envs/***/python3.7/site-packages/te/Lang/cce/te_compute/broadcast_compute.py", line 20, in <module>
        from .util import dtype_echeck_decorator
      File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/te_compute/util.py", line 22, in <module>
        from te.platform import intrinsic_check_support
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/__init__.py", line 38, in <module>
        from .cce_buffer import cur cce_product_params
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_buffer.py", line 20, in <module>
        from te.platform import cce_conf
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_conf.py", line 25, in <module>
        from te.platform import cce_build
      File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_build.py", line 201, in <module>
        pipline_opt_level=2
      ……
    tvm._ffi.base.TVMError: Traceback (most recent call last):
      ……
    TVMError: [EB0000] Buildconfig does hot contain field precise_bound_pipeline_opt level
    ……
    RuntimeError: execute_npu_operate: /usrl/workspace/PyTorch_Apex_Daily/CODE/aten/src/ATen/native/npu/utils/CalcuOpUtil .cpp:680 NPU error,NPU error code is:500001

原因分析

PyTorch内调用NPU类型算子时,强依赖于te、cce、TVM组件,PyTorch、CANN/nnae和te版本需要一致。在更新CANN/nnae后,te等组件不会自动更新,当版本不匹配时,则会出现该报错。

解决措施

更新te等组件版本,具体需要更新te-*.whl和topi-*.whl安装包。

在CANN安装目录或者nnae安装目录的lib64子目录下(以root安装用户为例:默认安装路径在/usr/local/Ascend/ascend-toolkit/latest/lib64目录下,更新安装包即可),找到te-*.whl和topi-*.whl安装包,使用pip3 install命令分别执行安装即可。

本页内容

该页面对您有帮助吗?
我要评分