在将自己的onnx转到om时报错，不知道是不是pth转onnx格式时出现问题，还是不支持某些算子。
以上是我将.pth文件转为.onnx文件的程序
错误报告如下：
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/miniconda3/envs/test/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
File "<string>", line 2, in get
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/managers.py", line 810, in _callmethod
kind, result = conn.recv()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
\
Traceback (most recent call last):
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [Conv_20,[Conv_20,Conv_20]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[Conv_20,[Conv_20,Conv_20]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[40]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_30,[Conv_30,Conv_30]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_30,[Conv_30,Conv_30]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[42]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_13,[Conv_13,Conv_13]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_13,[Conv_13,Conv_13]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[55]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_28,[Conv_28,Conv_28]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_28,[Conv_28,Conv_28]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[46]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_17,[Conv_17,Conv_17]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_17,[Conv_17,Conv_17]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[57]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_25,[Conv_25,Conv_25]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_25,[Conv_25,Conv_25]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[49]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084239441184] recompile single op[Conv_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084239441184] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_30] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_13] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_28] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_25] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084281711904] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph27[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph24[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
In op [fix_pipe], [fix_pipe does not support single op compilation!]
Failed to compile Op [Conv_9function_graph_3/Conv_9_pass.1,[Relu_10,Relu_10]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_9function_graph_3/Conv_9_pass.1,[Relu_10,Relu_10]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[76]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3function_graph_2/Conv_3_pass.1,[Relu_4,Relu_4]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_3function_graph_2/Conv_3_pass.1,[Relu_4,Relu_4]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[74]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0function_graph_0/Conv_0_pass.1,[Relu_1,Relu_1]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_0function_graph_0/Conv_0_pass.1,[Relu_1,Relu_1]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[78]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3function_graph_2/Conv_3,[Conv_3,Conv_3]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_3function_graph_2/Conv_3,[Conv_3,Conv_3]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[73]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_9function_graph_3/Conv_9,[Conv_9,Conv_9]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_9function_graph_3/Conv_9,[Conv_9,Conv_9]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[75]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0function_graph_0/Conv_0,[Conv_0,Conv_0]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_0function_graph_0/Conv_0,[Conv_0,Conv_0]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[77]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_6function_graph_1/Conv_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_0function_graph_0/Conv_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_9function_graph_3/Conv_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_3function_graph_2/Conv_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_6function_graph_1/Conv_6_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_0function_graph_0/Conv_0_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_3function_graph_2/Conv_3_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_9function_graph_3/Conv_9_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_17] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084256349472] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
点赞
收藏
回复
分享
举报
已解决
在将自己的onnx转到om时报错，不知道是不是pth转onnx格式时出现问题，还是不支持某些算子。
from modelDCAN import * # 设置使用CPU导出模型 device = torch.device("cpu") net = Net() net.load_state_dict(torch.load(r"./net_50.pth")) # 模型设置为推理模式 net.eval() input_data = torch.randn(1, 1, 256, 256) # 举例输入 # 导出模型到 ONNX 格式 torch.onnx.export(net, input_data, "DCAN.onnx", opset_version=11, input_names=['input'], output_names=['output'], dynamic_axes={"input":{0:"batch_size"}, "output":{0:"batch_size"}})以上是我将.pth文件转为.onnx文件的程序
错误报告如下：
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/miniconda3/envs/test/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
File "<string>", line 2, in get
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/managers.py", line 810, in _callmethod
kind, result = conn.recv()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
\
Traceback (most recent call last):
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 207, in finalize
cls.bank_process_pool.close()
AttributeError: 'NoneType' object has no attribute 'close'
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [Conv_20,[Conv_20,Conv_20]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[Conv_20,[Conv_20,Conv_20]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[40]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_30,[Conv_30,Conv_30]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_30,[Conv_30,Conv_30]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[42]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_13,[Conv_13,Conv_13]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_13,[Conv_13,Conv_13]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[55]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_28,[Conv_28,Conv_28]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_28,[Conv_28,Conv_28]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[46]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_17,[Conv_17,Conv_17]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_17,[Conv_17,Conv_17]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[57]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_25,[Conv_25,Conv_25]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
], optype: [Conv2D])
Compile op[Conv_25,[Conv_25,Conv_25]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[49]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084239441184] recompile single op[Conv_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084239441184] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_30] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_13] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_28] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084281711904] recompile single op[Conv_25] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084281711904] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank5_new_sub_graph27[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph24[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
In op [fix_pipe], [fix_pipe does not support single op compilation!]
Failed to compile Op [Conv_9function_graph_3/Conv_9_pass.1,[Relu_10,Relu_10]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_9function_graph_3/Conv_9_pass.1,[Relu_10,Relu_10]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[76]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3function_graph_2/Conv_3_pass.1,[Relu_4,Relu_4]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_3function_graph_2/Conv_3_pass.1,[Relu_4,Relu_4]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[74]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0function_graph_0/Conv_0_pass.1,[Relu_1,Relu_1]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/utils/errormgr/error_manager_vector.py", line 284, in raise_err_specific_reson
raise RuntimeError(args_dict, msg)
RuntimeError: ({'errCode': 'E61001', 'op_name': 'fix_pipe', 'reason': 'fix_pipe does not support single op compilation!'}, 'In op [fix_pipe], [fix_pipe does not support single op compilation!]')
], optype: [FixPipe])
Compile op[Conv_0function_graph_0/Conv_0_pass.1,[Relu_1,Relu_1]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/fix_pipe.py], optype[FixPipe], taskID[78]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_3function_graph_2/Conv_3,[Conv_3,Conv_3]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_3function_graph_2/Conv_3,[Conv_3,Conv_3]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[73]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_9function_graph_3/Conv_9,[Conv_9,Conv_9]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_9function_graph_3/Conv_9,[Conv_9,Conv_9]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[75]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
Failed to compile Op [Conv_0function_graph_0/Conv_0,[Conv_0,Conv_0]]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py failed with errormsg/stack: File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
], optype: [Conv2D])
Compile op[Conv_0function_graph_0/Conv_0,[Conv_0,Conv_0]] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[77]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_6function_graph_1/Conv_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_0function_graph_0/Conv_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_9function_graph_3/Conv_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_3function_graph_2/Conv_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_6function_graph_1/Conv_6_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_0function_graph_0/Conv_0_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_3function_graph_2/Conv_3_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_9function_graph_3/Conv_9_pass.1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[255084256349472] recompile single op[Conv_17] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
[SubGraphOpt][Compile][ParalCompOp] Thread[255084256349472] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
Compile operator failed, cause: 32Broken pipe File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
RouteServer.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 51, in wrapper
return func(cls, *args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 212, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 41, in finalize
self.__dynamic_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 325, in close
for name in copy.deepcopy(self.write_kb.keys()):
File "<string>", line 2, in keys
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/managers.py", line 809, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda3/envs/test/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)