【用户类型】：企业
【昇腾产品型号】：Atlas 200l DK A2 开发者套件 Ascend310B4
【版本信息】：
--操作系统版本：ubuntu.22.04 LTS aarch64
--固件驱动版本：npu-smi 23.0.rc3 Version:23.0.rc3
--CANN版本：CANN 7.0.rc1
--Python版本：Python3.9.2
【开发内容】：模型转换，使用ATC 将yolov5s.onnx转换为yolov5.om模型格式
【问题描述】：基于昇腾官网给的在线制卡流程将系统做好之后，使用atc进行模型转化，模型转换失败！
制卡参考网址：ttps://www.hiascend.com/document/detail/zh/Atlas200IDKA2DeveloperKit/23.0.RC2/qs/qs_0005.html
模型转换命令：atc --model=yolov5s.onnx --framework=5 --input_shape="images:1,3,640,640" --output=yolov5s --soc_version=Ascend310B4
请问该怎么解决呢？
【问题报错】
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
File "<string>", line 2, in get
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
kind, result = conn.recv()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
/
|
Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [/model.24/m.0/Conv,[/model.24/m.0/Conv,/model.24/m.0/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[/model.24/m.0/Conv,[/model.24/m.0/Conv,/model.24/m.0/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[272]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.24/m.1/Conv,[/model.24/m.1/Conv,/model.24/m.1/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[/model.24/m.1/Conv,[/model.24/m.1/Conv,/model.24/m.1/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[268]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.24/m.2/Conv,[/model.24/m.2/Conv,/model.24/m.2/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[/model.24/m.2/Conv,[/model.24/m.2/Conv,/model.24/m.2/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[264]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/cv1/conv/Conv,[/model.13/cv1/conv/Conv,/model.13/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/cv1/conv/Conv,[/model.13/cv1/conv/Conv,/model.13/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[343]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/m/m.0/cv1/conv/Conv,[/model.13/m/m.0/cv1/conv/Conv,/model.13/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/m/m.0/cv1/conv/Conv,[/model.13/m/m.0/cv1/conv/Conv,/model.13/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[349]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.14/conv/Conv,[/model.14/conv/Conv,/model.14/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.14/conv/Conv,[/model.14/conv/Conv,/model.14/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[352]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/m/m.0/cv2/conv/Conv,[/model.13/m/m.0/cv2/conv/Conv,/model.13/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/m/m.0/cv2/conv/Conv,[/model.13/m/m.0/cv2/conv/Conv,/model.13/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[355]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/cv3/conv/Conv,[/model.13/cv3/conv/Conv,/model.13/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/cv3/conv/Conv,[/model.13/cv3/conv/Conv,/model.13/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[358]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.14/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084700749088] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph124[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [/model.17/cv1/conv/Conv,[/model.17/cv1/conv/Conv,/model.17/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.17/cv1/conv/Conv,[/model.17/cv1/conv/Conv,/model.17/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[374]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.7/conv/Conv,[/model.7/conv/Conv,/model.7/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.7/conv/Conv,[/model.7/conv/Conv,/model.7/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[367]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.9/cv2/conv/Conv,[/model.9/cv2/conv/Conv,/model.9/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.9/cv2/conv/Conv,[/model.9/cv2/conv/Conv,/model.9/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[364]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.8/m/m.0/cv2/conv/Conv,[/model.8/m/m.0/cv2/conv/Conv,/model.8/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.8/m/m.0/cv2/conv/Conv,[/model.8/m/m.0/cv2/conv/Conv,/model.8/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[361]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/cv2/conv/Conv,[/model.4/cv2/conv/Conv,/model.4/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/cv2/conv/Conv,[/model.4/cv2/conv/Conv,/model.4/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[379]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.18/conv/Conv,[/model.18/conv/Conv,/model.18/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.18/conv/Conv,[/model.18/conv/Conv,/model.18/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[385]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/m/m.1/cv1/conv/Conv,[/model.4/m/m.1/cv1/conv/Conv,/model.4/m/m.1/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/m/m.1/cv1/conv/Conv,[/model.4/m/m.1/cv1/conv/Conv,/model.4/m/m.1/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[386]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/cv3/conv/Conv,[/model.23/cv3/conv/Conv,/model.23/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/cv3/conv/Conv,[/model.23/cv3/conv/Conv,/model.23/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[390]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/m/m.1/cv2/conv/Conv,[/model.4/m/m.1/cv2/conv/Conv,/model.4/m/m.1/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/m/m.1/cv2/conv/Conv,[/model.4/m/m.1/cv2/conv/Conv,/model.4/m/m.1/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[393]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.5/conv/Conv,[/model.5/conv/Conv,/model.5/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.5/conv/Conv,[/model.5/conv/Conv,/model.5/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[373]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.17/cv3/conv/Conv,[/model.17/cv3/conv/Conv,/model.17/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.17/cv3/conv/Conv,[/model.17/cv3/conv/Conv,/model.17/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[402]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.21/conv/Conv,[/model.21/conv/Conv,/model.21/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.21/conv/Conv,[/model.21/conv/Conv,/model.21/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[457]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/cv2/conv/Conv,[/model.23/cv2/conv/Conv,/model.23/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/cv2/conv/Conv,[/model.23/cv2/conv/Conv,/model.23/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[464]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/cv1/conv/Conv,[/model.2/cv1/conv/Conv,/model.2/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/cv1/conv/Conv,[/model.2/cv1/conv/Conv,/model.2/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[478]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/m/m.0/cv1/conv/Conv,[/model.2/m/m.0/cv1/conv/Conv,/model.2/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/m/m.0/cv1/conv/Conv,[/model.2/m/m.0/cv1/conv/Conv,/model.2/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[484]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.1/conv/Conv,[/model.1/conv/Conv,/model.1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.1/conv/Conv,[/model.1/conv/Conv,/model.1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[496]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.0/conv/Conv,[/model.0/conv/Conv,/model.0/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.0/conv/Conv,[/model.0/conv/Conv,/model.0/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[499]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/m/m.0/cv1/conv/Conv,[/model.23/m/m.0/cv1/conv/Conv,/model.23/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/m/m.0/cv1/conv/Conv,[/model.23/m/m.0/cv1/conv/Conv,/model.23/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[396]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.18/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.1/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.21/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.0/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.2/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084331912480] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Failed to compile Op [/model.2/m/m.0/cv2/conv/Conv,[/model.2/m/m.0/cv2/conv/Conv,/model.2/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/m/m.0/cv2/conv/Conv,[/model.2/m/m.0/cv2/conv/Conv,/model.2/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[505]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph126[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [/model.3/conv/Conv,[/model.3/conv/Conv,/model.3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.3/conv/Conv,[/model.3/conv/Conv,/model.3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[511]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/cv3/conv/Conv,[/model.2/cv3/conv/Conv,/model.2/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/cv3/conv/Conv,[/model.2/cv3/conv/Conv,/model.2/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[508]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.10/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.9/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.5/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.1/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.1/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.7/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.0/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.1/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.2/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.9/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.1/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.2/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084348820768] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 808, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
点赞
收藏
回复
分享
举报
已解决
【用户类型】：企业
【昇腾产品型号】：Atlas 200l DK A2 开发者套件 Ascend310B4
【版本信息】：
--操作系统版本：ubuntu.22.04 LTS aarch64
--固件驱动版本：npu-smi 23.0.rc3 Version:23.0.rc3
--CANN版本：CANN 7.0.rc1
--Python版本：Python3.9.2
【开发内容】：模型转换，使用ATC 将yolov5s.onnx转换为yolov5.om模型格式
【问题描述】：基于昇腾官网给的在线制卡流程将系统做好之后，使用atc进行模型转化，模型转换失败！
制卡参考网址：ttps://www.hiascend.com/document/detail/zh/Atlas200IDKA2DeveloperKit/23.0.RC2/qs/qs_0005.html
模型转换命令：atc --model=yolov5s.onnx --framework=5 --input_shape="images:1,3,640,640" --output=yolov5s --soc_version=Ascend310B4
请问该怎么解决呢？
【问题报错】
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
File "<string>", line 2, in get
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
kind, result = conn.recv()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
/
|
Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [/model.24/m.0/Conv,[/model.24/m.0/Conv,/model.24/m.0/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[/model.24/m.0/Conv,[/model.24/m.0/Conv,/model.24/m.0/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[272]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.24/m.1/Conv,[/model.24/m.1/Conv,/model.24/m.1/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[/model.24/m.1/Conv,[/model.24/m.1/Conv,/model.24/m.1/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[268]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.24/m.2/Conv,[/model.24/m.2/Conv,/model.24/m.2/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[/model.24/m.2/Conv,[/model.24/m.2/Conv,/model.24/m.2/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[264]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/cv1/conv/Conv,[/model.13/cv1/conv/Conv,/model.13/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/cv1/conv/Conv,[/model.13/cv1/conv/Conv,/model.13/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[343]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/m/m.0/cv1/conv/Conv,[/model.13/m/m.0/cv1/conv/Conv,/model.13/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/m/m.0/cv1/conv/Conv,[/model.13/m/m.0/cv1/conv/Conv,/model.13/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[349]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.14/conv/Conv,[/model.14/conv/Conv,/model.14/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.14/conv/Conv,[/model.14/conv/Conv,/model.14/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[352]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/m/m.0/cv2/conv/Conv,[/model.13/m/m.0/cv2/conv/Conv,/model.13/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/m/m.0/cv2/conv/Conv,[/model.13/m/m.0/cv2/conv/Conv,/model.13/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[355]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.13/cv3/conv/Conv,[/model.13/cv3/conv/Conv,/model.13/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.13/cv3/conv/Conv,[/model.13/cv3/conv/Conv,/model.13/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[358]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.14/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084700749088] recompile single op[/model.13/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084700749088] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph124[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [/model.17/cv1/conv/Conv,[/model.17/cv1/conv/Conv,/model.17/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.17/cv1/conv/Conv,[/model.17/cv1/conv/Conv,/model.17/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[374]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.7/conv/Conv,[/model.7/conv/Conv,/model.7/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.7/conv/Conv,[/model.7/conv/Conv,/model.7/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[367]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.9/cv2/conv/Conv,[/model.9/cv2/conv/Conv,/model.9/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.9/cv2/conv/Conv,[/model.9/cv2/conv/Conv,/model.9/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[364]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.8/m/m.0/cv2/conv/Conv,[/model.8/m/m.0/cv2/conv/Conv,/model.8/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.8/m/m.0/cv2/conv/Conv,[/model.8/m/m.0/cv2/conv/Conv,/model.8/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[361]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/cv2/conv/Conv,[/model.4/cv2/conv/Conv,/model.4/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/cv2/conv/Conv,[/model.4/cv2/conv/Conv,/model.4/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[379]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.18/conv/Conv,[/model.18/conv/Conv,/model.18/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.18/conv/Conv,[/model.18/conv/Conv,/model.18/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[385]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/m/m.1/cv1/conv/Conv,[/model.4/m/m.1/cv1/conv/Conv,/model.4/m/m.1/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/m/m.1/cv1/conv/Conv,[/model.4/m/m.1/cv1/conv/Conv,/model.4/m/m.1/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[386]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/cv3/conv/Conv,[/model.23/cv3/conv/Conv,/model.23/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/cv3/conv/Conv,[/model.23/cv3/conv/Conv,/model.23/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[390]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.4/m/m.1/cv2/conv/Conv,[/model.4/m/m.1/cv2/conv/Conv,/model.4/m/m.1/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.4/m/m.1/cv2/conv/Conv,[/model.4/m/m.1/cv2/conv/Conv,/model.4/m/m.1/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[393]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.5/conv/Conv,[/model.5/conv/Conv,/model.5/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.5/conv/Conv,[/model.5/conv/Conv,/model.5/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[373]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.17/cv3/conv/Conv,[/model.17/cv3/conv/Conv,/model.17/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.17/cv3/conv/Conv,[/model.17/cv3/conv/Conv,/model.17/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[402]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.21/conv/Conv,[/model.21/conv/Conv,/model.21/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.21/conv/Conv,[/model.21/conv/Conv,/model.21/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[457]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/cv2/conv/Conv,[/model.23/cv2/conv/Conv,/model.23/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/cv2/conv/Conv,[/model.23/cv2/conv/Conv,/model.23/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[464]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/cv1/conv/Conv,[/model.2/cv1/conv/Conv,/model.2/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/cv1/conv/Conv,[/model.2/cv1/conv/Conv,/model.2/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[478]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/m/m.0/cv1/conv/Conv,[/model.2/m/m.0/cv1/conv/Conv,/model.2/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/m/m.0/cv1/conv/Conv,[/model.2/m/m.0/cv1/conv/Conv,/model.2/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[484]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.1/conv/Conv,[/model.1/conv/Conv,/model.1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.1/conv/Conv,[/model.1/conv/Conv,/model.1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[496]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.0/conv/Conv,[/model.0/conv/Conv,/model.0/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.0/conv/Conv,[/model.0/conv/Conv,/model.0/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[499]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.23/m/m.0/cv1/conv/Conv,[/model.23/m/m.0/cv1/conv/Conv,/model.23/m/m.0/cv1/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.23/m/m.0/cv1/conv/Conv,[/model.23/m/m.0/cv1/conv/Conv,/model.23/m/m.0/cv1/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[396]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.18/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.17/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.23/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.1/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.21/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.0/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.24/m.2/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084331912480] recompile single op[/model.20/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084331912480] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Failed to compile Op [/model.2/m/m.0/cv2/conv/Conv,[/model.2/m/m.0/cv2/conv/Conv,/model.2/m/m.0/cv2/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/m/m.0/cv2/conv/Conv,[/model.2/m/m.0/cv2/conv/Conv,/model.2/m/m.0/cv2/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[505]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph126[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [/model.3/conv/Conv,[/model.3/conv/Conv,/model.3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.3/conv/Conv,[/model.3/conv/Conv,/model.3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[511]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [/model.2/cv3/conv/Conv,[/model.2/cv3/conv/Conv,/model.2/cv3/conv/Conv]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[/model.2/cv3/conv/Conv,[/model.2/cv3/conv/Conv,/model.2/cv3/conv/Conv]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[508]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.10/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.9/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.5/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.1/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/m/m.1/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.4/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.8/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.7/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.2/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.0/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.0/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.1/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.2/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.9/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv2/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.0/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.1/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/cv3/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084348820768] recompile single op[/model.6/m/m.2/cv1/conv/Conv] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084348820768] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 808, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)