虚拟机atc工具转换模型失败解决方法
收藏回复举报
虚拟机atc工具转换模型失败解决方法
t('forum.solved') 已解决
发表于2024-06-06 16:17:10
0 查看
因在Altas200 DK A2的4G版本上转换模型耗时较长并且有会假死宕机,按照https://www.hiascend.com/document/detail/zh/Atlas200IDKA2DeveloperKit/23.0.RC2/Appendices/faq/faq_0029.html设置虚拟内存上限还是不能解决问题,所以有在虚拟机转换模型的需求,虚拟机镜像使用ubuntu-20.04.6-desktop-amd64.iso,CANN版本为社区版Ascend-cann-toolkit_7.0.1_linux-x86_64,按照安装步骤安装好CANN后使用atc指令转换模型
atc --model=yolov5s.onnx --framework=5 --output=model --soc_version=Ascend310B4

运行报错

lsn@lsn:~/infer_project$ atc --model=yolov5s.onnx --framework=5 --output=model --soc_version=Ascend310B4
ATC start working now, please wait for a moment.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
    key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
  File "<string>", line 2, in get
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 835, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
    return func(cls, *args, **kwargs)
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
    cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [Conv_272,[Conv_272,Conv_272]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Solution: See the host log for details, and then check the Python stack where the error log is reported.
        TraceBack (most recent call last):
        Compile op[Conv_272,[Conv_272,Conv_272]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[270]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_235,[Conv_235,Conv_235]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_235,[Conv_235,Conv_235]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[274]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_198,[Conv_198,Conv_198]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_198,[Conv_198,Conv_198]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[278]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_133,[Conv_133,Conv_133]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_133,[Conv_133,Conv_133]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[350]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_123,[Conv_123,Conv_123]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_123,[Conv_123,Conv_123]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[353]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_126,[Conv_126,Conv_126]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_126,[Conv_126,Conv_126]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[356]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_136,[Conv_136,Conv_136]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_136,[Conv_136,Conv_136]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[365]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_120,[Conv_120,Conv_120]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_120,[Conv_120,Conv_120]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[359]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_84,[Conv_84,Conv_84]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_84,[Conv_84,Conv_84]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[368]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_26,[Conv_26,Conv_26]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_26,[Conv_26,Conv_26]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[371]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_111,[Conv_111,Conv_111]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_111,[Conv_111,Conv_111]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[377]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_50,[Conv_50,Conv_50]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_50,[Conv_50,Conv_50]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[380]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_36,[Conv_36,Conv_36]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_36,[Conv_36,Conv_36]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[383]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_39,[Conv_39,Conv_39]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_39,[Conv_39,Conv_39]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[386]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_129] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_120] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_136] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_126] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_123] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026226648832] recompile single op[Conv_133] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140026226648832] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1001]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1127]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1292]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph133[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
        Failed to compile Op [Conv_47,[Conv_47,Conv_47]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_47,[Conv_47,Conv_47]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[422]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_188,[Conv_188,Conv_188]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_188,[Conv_188,Conv_188]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[428]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_185,[Conv_185,Conv_185]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_185,[Conv_185,Conv_185]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[431]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_195,[Conv_195,Conv_195]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_195,[Conv_195,Conv_195]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[425]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_178,[Conv_178,Conv_178]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_178,[Conv_178,Conv_178]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[434]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_182,[Conv_182,Conv_182]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_182,[Conv_182,Conv_182]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[437]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_16,[Conv_16,Conv_16]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_16,[Conv_16,Conv_16]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[448]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_9,[Conv_9,Conv_9]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_9,[Conv_9,Conv_9]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[454]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_0,[Conv_0,Conv_0]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_0,[Conv_0,Conv_0]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[464]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_158,[Conv_158,Conv_158]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_158,[Conv_158,Conv_158]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[471]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_12,[Conv_12,Conv_12]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_12,[Conv_12,Conv_12]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[496]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_3,[Conv_3,Conv_3]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_3,[Conv_3,Conv_3]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[490]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_20,[Conv_20,Conv_20]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_20,[Conv_20,Conv_20]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[499]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        Failed to compile Op [Conv_142,[Conv_142,Conv_142]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_142,[Conv_142,Conv_142]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[504]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_151] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_142] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_158] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_191] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_182] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_185] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_188] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_178] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_155] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_148] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_145] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_235] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_272] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_175] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_165] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_195] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_198] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_168] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_162] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140026251826944] recompile single op[Conv_171] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140026251826944] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1001]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph135[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
        Failed to compile Op [Conv_23,[Conv_23,Conv_23]]. (oppath: [Compile /home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
        Compile op[Conv_23,[Conv_23,Conv_23]] failed, oppath[/home/lsn/Ascend/ascend-toolkit/7.0.1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[516]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:771]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_12] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_101] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_90] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_93] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_47] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_32] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_104] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_39] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_29] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_87] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_36] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_97] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_50] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_111] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_43] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_26] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_84] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_77] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_53] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_59] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_66] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_73] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_56] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_23] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_63] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_70] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_114] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140025572345600] recompile single op[Conv_81] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:954]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140025572345600] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1001]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
        subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:996]
        build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
        GenerateOfflineModel execute failed.
        Compile operator failed, cause: 32Broken pipe   File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
    RouteServer.finalize()
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
    return func(cls, *args, **kwargs)
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
    cls.cann_kb_mgr.finalize()
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
    self.__dynamic_mgr.close()
  File "/home/lsn/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 327, in close
    for name in copy.deepcopy(self.write_kb.keys()):
  File "<string>", line 2, in keys
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 834, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)

将虚拟机内存由8G增加到16G问题解决,可以正常转换模型

cke_10461.png

我要发帖子