ATC start working now, please wait for a moment.
...
ATC run failed, Please check the detail log, Try 'atc --help' for more information
E40021: Failed to compile Op [/enc_p/encoder/attn_layers.0/Softmax]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/softmax_v2.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/python/site-packages/tbe/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
raise get_last_ffi_error()
tvm._ffi.base.TVMError: {'errClass': 'Storage Error', 'errCode': '[EB3000]', 'message': 'Traceback (most recent call last):
19: TVMFuncCall
18: ascend_tvm::runtime::TypedPackedFunc<ascend_tvm::IRModule (ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)>::AssignTypedLambda<ascend_tvm::{lambda(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)#5}>(ascend_tvm::{lambda(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)#5}, std::string)::{lambda(ascend_tvm::runtime::TVMArgs const&, ascend_tvm::runtime::TVMRetValue*)#1}::operator()(ascend_tvm::runtime::TVMArgs const, ascend_tvm::runtime::TVMRetValue) const
17: ascend_tvm::LowerSchedule(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, std::string const&, std::unordered_map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, std::hash<ascend_tvm::te::Tensor>, std::equal_to<ascend_tvm::te::Tensor>, std::allocator<std::pair<ascend_tvm::te::Tensor const, ascend_tvm::tir::Buffer> > > const&, ascend_tvm::PrimExpr, bool)
16: ascend_tvm::LowerWithPassList(ascend_tvm::IRModule, ascend_tvm::runtime::Array<ascend_tvm::transform::Pass, void>)
15: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule) const
14: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const
13: ascend_tvm::transform::SequentialNode::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const
12: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const
11: ascend_tvm::tir::transform::PrimFuncPassNode::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const
10: std::_Function_handler<void (ascend_tvm::runtime::TVMArgs, ascend_tvm::runtime::TVMRetValue*), ascend_tvm::runtime::TypedPackedFunc<ascend_tvm::tir::PrimFunc (ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)>::AssignTypedLambda<ascend_tvm::tir::transform::StorageRewriteCCE(ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&)::{lambda(ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)#1}>(ascend_tvm::tir::transform::StorageRewriteCCE(ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&)::{lambda(ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)#1})::{lambda(ascend_tvm::runtime::TVMArgs const&, ascend_tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, ascend_tvm::runtime::TVMArgs&&, ascend_tvm::runtime::TVMRetValue*&&)
9: ascend_tvm::tir::StorageRewriteCCE(ascend_tvm::tir::Stmt, ascend_tvm::tir::StorageRewriteCCEConfig const&, ascend_tvm::tir::PrimFuncAttrs const&)
8: ascend_tvm::tir::storage_utils::DoStorageRewrite(ascend_tvm::tir::Stmt, std::unordered_map<ascend_tvm::tir::VarNode const*, ascend_tvm::tir::storage_utils::BufferEntry, std::hash<ascend_tvm::tir::VarNode const*>, std::equal_to<ascend_tvm::tir::VarNode const*>, std::allocator<std::pair<ascend_tvm::tir::VarNode const* const, ascend_tvm::tir::storage_utils::BufferEntry> > >, ascend_tvm::tir::StorageRewriteCCEConfig, ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&, bool&)
7: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::Rewrite(ascend_tvm::tir::Stmt, std::unordered_map<ascend_tvm::tir::VarNode const*, ascend_tvm::tir::storage_utils::BufferEntry, std::hash<ascend_tvm::tir::VarNode const*>, std::equal_to<ascend_tvm::tir::VarNode const*>, std::allocator<std::pair<ascend_tvm::tir::VarNode const* const, ascend_tvm::tir::storage_utils::BufferEntry> > >, bool, ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&)
6: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::PrepareNewAlloc()
5: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::DoNewAlloc(std::vector<ascend_tvm::tir::storage_utils::StorageEntry*, std::allocator<ascend_tvm::tir::storage_utils::StorageEntry*> >&)
4: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::NewAllocTagMerged(ascend_tvm::tir::storage_utils::StorageEntry*, unsigned long, unsigned long const&)
3: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::ApplyFailStrategy(ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::StatusWrapper&, bool&, unsigned long const&, unsigned long)
2: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::GetAllocStatus(ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::StatusWrapper const&)
File "storage_rewrite_cce.cc", line 1323
TVMError: [EB3000] Allocation exceed bound of memory tag:local.UBLogFatal::Entry::Finalize()', 'errPcause': 'N/A', 'errSolution': 'N/A'}
], optype: [SoftmaxV2])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[/enc_p/encoder/attn_layers.0/Softmax] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/softmax_v2.py], optype[SoftmaxV2], taskID[176]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:764]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656647317248] recompile single op[/enc_p/encoder/attn_layers.0/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140656647317248] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1125]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank10_new_sub_graph280[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
subgraph 9 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:974]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658014672640] recompile single op[/enc_p/encoder/attn_layers.1/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140658014672640] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank44_new_sub_graph320[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656638924544] recompile single op[/enc_p/encoder/attn_layers.4/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140656638924544] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank119_new_sub_graph395[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658006279936] recompile single op[/enc_p/encoder/attn_layers.5/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140658006279936] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank144_new_sub_graph420[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658023065344] recompile single op[/enc_p/encoder/attn_layers.2/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140658023065344] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank69_new_sub_graph345[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656672495360] recompile single op[/enc_p/encoder/attn_layers.3/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971]
[SubGraphOpt][Compile][ParalCompOp] Thread[140656672495360] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017]
Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank94_new_sub_graph370[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
GenerateOfflineModel execute failed.
点赞
收藏
回复
分享
举报
已解决
环境
问题描述
使用以下命令转换模型：
会出现以下错误（如果把输入长度150改为100,就不会有错误）：
ATC start working now, please wait for a moment. ... ATC run failed, Please check the detail log, Try 'atc --help' for more information E40021: Failed to compile Op [/enc_p/encoder/attn_layers.0/Softmax]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/softmax_v2.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/python/site-packages/tbe/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__ raise get_last_ffi_error() tvm._ffi.base.TVMError: {'errClass': 'Storage Error', 'errCode': '[EB3000]', 'message': 'Traceback (most recent call last): 19: TVMFuncCall 18: ascend_tvm::runtime::TypedPackedFunc<ascend_tvm::IRModule (ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)>::AssignTypedLambda<ascend_tvm::{lambda(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)#5}>(ascend_tvm::{lambda(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, ascend_tvm::runtime::String const&, ascend_tvm::runtime::Map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, void, void> const&, ascend_tvm::runtime::ObjectRef, bool)#5}, std::string)::{lambda(ascend_tvm::runtime::TVMArgs const&, ascend_tvm::runtime::TVMRetValue*)#1}::operator()(ascend_tvm::runtime::TVMArgs const, ascend_tvm::runtime::TVMRetValue) const 17: ascend_tvm::LowerSchedule(ascend_tvm::te::Schedule, ascend_tvm::runtime::Array<ascend_tvm::runtime::ObjectRef, void> const&, std::string const&, std::unordered_map<ascend_tvm::te::Tensor, ascend_tvm::tir::Buffer, std::hash<ascend_tvm::te::Tensor>, std::equal_to<ascend_tvm::te::Tensor>, std::allocator<std::pair<ascend_tvm::te::Tensor const, ascend_tvm::tir::Buffer> > > const&, ascend_tvm::PrimExpr, bool) 16: ascend_tvm::LowerWithPassList(ascend_tvm::IRModule, ascend_tvm::runtime::Array<ascend_tvm::transform::Pass, void>) 15: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule) const 14: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const 13: ascend_tvm::transform::SequentialNode::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const 12: ascend_tvm::transform::Pass::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const 11: ascend_tvm::tir::transform::PrimFuncPassNode::operator()(ascend_tvm::IRModule, ascend_tvm::transform::PassContext const&) const 10: std::_Function_handler<void (ascend_tvm::runtime::TVMArgs, ascend_tvm::runtime::TVMRetValue*), ascend_tvm::runtime::TypedPackedFunc<ascend_tvm::tir::PrimFunc (ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)>::AssignTypedLambda<ascend_tvm::tir::transform::StorageRewriteCCE(ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&)::{lambda(ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)#1}>(ascend_tvm::tir::transform::StorageRewriteCCE(ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&)::{lambda(ascend_tvm::tir::PrimFunc, ascend_tvm::IRModule, ascend_tvm::transform::PassContext)#1})::{lambda(ascend_tvm::runtime::TVMArgs const&, ascend_tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, ascend_tvm::runtime::TVMArgs&&, ascend_tvm::runtime::TVMRetValue*&&) 9: ascend_tvm::tir::StorageRewriteCCE(ascend_tvm::tir::Stmt, ascend_tvm::tir::StorageRewriteCCEConfig const&, ascend_tvm::tir::PrimFuncAttrs const&) 8: ascend_tvm::tir::storage_utils::DoStorageRewrite(ascend_tvm::tir::Stmt, std::unordered_map<ascend_tvm::tir::VarNode const*, ascend_tvm::tir::storage_utils::BufferEntry, std::hash<ascend_tvm::tir::VarNode const*>, std::equal_to<ascend_tvm::tir::VarNode const*>, std::allocator<std::pair<ascend_tvm::tir::VarNode const* const, ascend_tvm::tir::storage_utils::BufferEntry> > >, ascend_tvm::tir::StorageRewriteCCEConfig, ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&, bool&) 7: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::Rewrite(ascend_tvm::tir::Stmt, std::unordered_map<ascend_tvm::tir::VarNode const*, ascend_tvm::tir::storage_utils::BufferEntry, std::hash<ascend_tvm::tir::VarNode const*>, std::equal_to<ascend_tvm::tir::VarNode const*>, std::allocator<std::pair<ascend_tvm::tir::VarNode const* const, ascend_tvm::tir::storage_utils::BufferEntry> > >, bool, ascend_tvm::runtime::Array<ascend_tvm::tir::CCEPrimFunc, void> const&) 6: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::PrepareNewAlloc() 5: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::DoNewAlloc(std::vector<ascend_tvm::tir::storage_utils::StorageEntry*, std::allocator<ascend_tvm::tir::storage_utils::StorageEntry*> >&) 4: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::NewAllocTagMerged(ascend_tvm::tir::storage_utils::StorageEntry*, unsigned long, unsigned long const&) 3: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::ApplyFailStrategy(ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::StatusWrapper&, bool&, unsigned long const&, unsigned long) 2: ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::GetAllocStatus(ascend_tvm::tir::storage_utils::StoragePlanRewriterCCE::StatusWrapper const&) File "storage_rewrite_cce.cc", line 1323 TVMError: [EB3000] Allocation exceed bound of memory tag:local.UBLogFatal::Entry::Finalize()', 'errPcause': 'N/A', 'errSolution': 'N/A'} ], optype: [SoftmaxV2]) Solution: See the host log for details, and then check the Python stack where the error log is reported. TraceBack (most recent call last): Compile op[/enc_p/encoder/attn_layers.0/Softmax] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/softmax_v2.py], optype[SoftmaxV2], taskID[176]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:764] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656647317248] recompile single op[/enc_p/encoder/attn_layers.0/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140656647317248] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1125] [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank10_new_sub_graph280[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] subgraph 9 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:974] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658014672640] recompile single op[/enc_p/encoder/attn_layers.1/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140658014672640] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank44_new_sub_graph320[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656638924544] recompile single op[/enc_p/encoder/attn_layers.4/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140656638924544] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank119_new_sub_graph395[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658006279936] recompile single op[/enc_p/encoder/attn_layers.5/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140658006279936] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank144_new_sub_graph420[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140658023065344] recompile single op[/enc_p/encoder/attn_layers.2/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140658023065344] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank69_new_sub_graph345[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140656672495360] recompile single op[/enc_p/encoder/attn_layers.3/Softmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:971] [SubGraphOpt][Compile][ParalCompOp] Thread[140656672495360] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1017] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank94_new_sub_graph370[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615] GenerateOfflineModel execute failed.