Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [Conv_198,[Conv_198,Conv_198]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[Conv_198,[Conv_198,Conv_198]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[277]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_216,[Conv_216,Conv_216]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_216,[Conv_216,Conv_216]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[273]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_234,[Conv_234,Conv_234]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_234,[Conv_234,Conv_234]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[269]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_120,[Conv_120,Conv_120]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_120,[Conv_120,Conv_120]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[348]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_133,[Conv_133,Conv_133]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_133,[Conv_133,Conv_133]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[354]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_126,[Conv_126,Conv_126]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_126,[Conv_126,Conv_126]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[357]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_136,[Conv_136,Conv_136]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_136,[Conv_136,Conv_136]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[360]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_123,[Conv_123,Conv_123]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_123,[Conv_123,Conv_123]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[363]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_123] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_136] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_126] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_133] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_129] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_120] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083903896864] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph133[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [Conv_188,[Conv_188,Conv_188]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_188,[Conv_188,Conv_188]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[366]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_9,[Conv_9,Conv_9]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_9,[Conv_9,Conv_9]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[367]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0,[Conv_0,Conv_0]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_0,[Conv_0,Conv_0]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[373]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_155,[Conv_155,Conv_155]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_155,[Conv_155,Conv_155]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[378]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_158,[Conv_158,Conv_158]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_158,[Conv_158,Conv_158]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[372]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_195,[Conv_195,Conv_195]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_195,[Conv_195,Conv_195]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[384]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_84,[Conv_84,Conv_84]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_84,[Conv_84,Conv_84]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[385]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_111,[Conv_111,Conv_111]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_111,[Conv_111,Conv_111]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[390]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_39,[Conv_39,Conv_39]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_39,[Conv_39,Conv_39]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[396]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_182,[Conv_182,Conv_182]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_182,[Conv_182,Conv_182]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[391]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_36,[Conv_36,Conv_36]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_36,[Conv_36,Conv_36]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[402]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_185,[Conv_185,Conv_185]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_185,[Conv_185,Conv_185]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[403]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_142,[Conv_142,Conv_142]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_142,[Conv_142,Conv_142]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[445]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_178,[Conv_178,Conv_178]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_178,[Conv_178,Conv_178]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[468]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_16,[Conv_16,Conv_16]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_16,[Conv_16,Conv_16]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[486]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3,[Conv_3,Conv_3]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_3,[Conv_3,Conv_3]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[495]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_20,[Conv_20,Conv_20]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_20,[Conv_20,Conv_20]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[504]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_23,[Conv_23,Conv_23]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_23,[Conv_23,Conv_23]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[507]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_43,[Conv_43,Conv_43]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_43,[Conv_43,Conv_43]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[513]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_12,[Conv_12,Conv_12]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_12,[Conv_12,Conv_12]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[510]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_50,[Conv_50,Conv_50]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_50,[Conv_50,Conv_50]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[519]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_178] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_151] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_142] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_185] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_191] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_182] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_158] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_155] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_188] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_195] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_234] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_216] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_175] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_148] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_198] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_145] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_168] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_162] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_171] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_165] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083786587424] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph135[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_50] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_12] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_43] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_23] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_90] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_29] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_36] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_114] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_104] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_87] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_97] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_32] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_39] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_111] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_101] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_93] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_56] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_63] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_26] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_70] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_77] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_84] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_53] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_47] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_59] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_66] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_73] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_81] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083912351008] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 808, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
点赞
收藏
回复
分享
举报
已解决
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [Conv_198,[Conv_198,Conv_198]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[Conv_198,[Conv_198,Conv_198]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[277]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_216,[Conv_216,Conv_216]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_216,[Conv_216,Conv_216]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[273]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_234,[Conv_234,Conv_234]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_234,[Conv_234,Conv_234]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[269]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_120,[Conv_120,Conv_120]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_120,[Conv_120,Conv_120]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[348]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_133,[Conv_133,Conv_133]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_133,[Conv_133,Conv_133]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[354]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_126,[Conv_126,Conv_126]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_126,[Conv_126,Conv_126]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[357]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_136,[Conv_136,Conv_136]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_136,[Conv_136,Conv_136]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[360]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_123,[Conv_123,Conv_123]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_123,[Conv_123,Conv_123]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[363]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_123] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_136] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_126] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_133] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_129] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083903896864] recompile single op[Conv_120] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083903896864] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph133[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Failed to compile Op [Conv_188,[Conv_188,Conv_188]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_188,[Conv_188,Conv_188]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[366]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_9,[Conv_9,Conv_9]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_9,[Conv_9,Conv_9]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[367]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0,[Conv_0,Conv_0]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_0,[Conv_0,Conv_0]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[373]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_155,[Conv_155,Conv_155]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_155,[Conv_155,Conv_155]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[378]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_158,[Conv_158,Conv_158]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_158,[Conv_158,Conv_158]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[372]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_195,[Conv_195,Conv_195]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_195,[Conv_195,Conv_195]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[384]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_84,[Conv_84,Conv_84]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_84,[Conv_84,Conv_84]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[385]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_111,[Conv_111,Conv_111]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_111,[Conv_111,Conv_111]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[390]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_39,[Conv_39,Conv_39]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_39,[Conv_39,Conv_39]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[396]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_182,[Conv_182,Conv_182]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_182,[Conv_182,Conv_182]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[391]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_36,[Conv_36,Conv_36]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_36,[Conv_36,Conv_36]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[402]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_185,[Conv_185,Conv_185]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_185,[Conv_185,Conv_185]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[403]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_142,[Conv_142,Conv_142]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_142,[Conv_142,Conv_142]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[445]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_178,[Conv_178,Conv_178]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_178,[Conv_178,Conv_178]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[468]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_16,[Conv_16,Conv_16]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_16,[Conv_16,Conv_16]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[486]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3,[Conv_3,Conv_3]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_3,[Conv_3,Conv_3]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[495]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_20,[Conv_20,Conv_20]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_20,[Conv_20,Conv_20]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[504]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_23,[Conv_23,Conv_23]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_23,[Conv_23,Conv_23]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[507]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_43,[Conv_43,Conv_43]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_43,[Conv_43,Conv_43]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[513]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_12,[Conv_12,Conv_12]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_12,[Conv_12,Conv_12]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[510]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_50,[Conv_50,Conv_50]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_50,[Conv_50,Conv_50]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[519]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_178] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_151] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_142] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_185] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_191] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_182] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_158] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_155] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_188] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_195] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_234] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_216] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_175] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_148] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_198] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_145] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_168] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_162] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_171] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083786587424] recompile single op[Conv_165] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083786587424] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph135[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_50] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_12] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_43] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_23] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_90] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_29] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_36] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_114] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_104] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_87] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_97] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_32] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_39] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_111] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_101] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_93] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_56] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_63] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_26] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_70] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_77] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_84] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_53] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_47] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_59] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_66] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_73] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255083912351008] recompile single op[Conv_81] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255083912351008] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/managers.py", line 808, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/lib/python3.9/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)