2023-08-02 02:32:59,352 - mmdet - INFO - workflow: [('train', 1)], max: 10 epochs 

2023-08-02 02:32:59,380 - mmdet - INFO - Checkpoints will be saved to /media/sda/shengteng_test/ModelZoo-PyTorch/PyTorch/dev/cv/detection/YOLOX_ID2833_for_PyTorch/work_dirs/yolox_m_8(eight)x8(eight)_300e_ 

coco by HardDiskBackend. 

- 

    raise get_last_ffi_error() 

tvm._ffi.base.TVMError: {'errClass': 'Inner Error', 'errCode': '[EB9000]', 'message': 'Traceback (most recent call last): 

  7: TVMFuncCall 

  6: std::_Function_handler<void (ascend_tvm::runtime::TVMArgs, ascend_tvm::runtime::TVMRetValue*), ascend_tvm::te::{lambda(ascend_tvm::runtime::TVMArgs, ascend_tvm::runtime::TVMRetValue*)#3 

0}>::_M_invoke(std::_Any_data const&, ascend_tvm::runtime::TVMArgs&&, ascend_tvm::runtime::TVMRetValue*&&) 

  5: ascend_tvm::te::Schedule::rfactor(ascend_tvm::te::Tensor const&, ascend_tvm::runtime::Array<ascend_tvm::tir::IterVar, void> const&, ascend_tvm::runtime::Array<ascend_tvm::Integer, void> 

 const&, std::string) 

  4: ascend_tvm::te::MakeReplRedAxis(ascend_tvm::runtime::Array<ascend_tvm::tir::IterVar, void> const&, std::unordered_map<ascend_tvm::tir::IterVar, ascend_tvm::Range, std::hash<ascend_tvm:: 

tir::IterVar>, std::equal_to<ascend_tvm::tir::IterVar>, std::allocator<std::pair<ascend_tvm::tir::IterVar const, ascend_tvm::Range> > > const&, ascend_tvm::te::NormalModeParam) 

  3: ascend_tvm::te::reduce_axis(ascend_tvm::Range, std::string) 

  2: ascend_tvm::Range::type() 

  1: ascend_tvm::runtime::detail::LogFatal::Entry::Finalize() 

  0: ascend_tvm::runtime::Backtrace() 

  File "expr.cc", line 154 

 

TVMError: [EB9000] Check failed: (r->min.dtype() == r->extent.dtype()) is false: Region min and extent must have same type ', 'errPcause': 'N/A', 'errSolution': 'N/A'} 

], optype: [BNTrainingReduce]) 

        Solution: See the host log for details, and then check the Python stack where the error log is reported. 

        TraceBack (most recent call last): 

        Compile op[BNTrainingReduce] failed, oppath[/usr/local/Ascend/ascend-toolkit/6.3.RC2.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/bn_training_reduce.py], optype[BNTrainingR 

educe], taskID[19]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:855] 

        Compile op[BNTrainingReduce] failed, oppath[/usr/local/Ascend/ascend-toolkit/6.3.RC2.alpha003/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/bn_training_reduce.py], optype[BNTrainingR 

educe], taskID[20]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:855] 

        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281458048487904] recompile single op[BNTrainingReduce] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE: 

891] 

        [SubGraphOpt][Compile][ParalCompOp] Thread[281458048487904] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:937] 

        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1127] 

        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1306] 

        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131] 

        subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:770] 

        build graph failed, graph id:5, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1492] 

        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] 

        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] 

 

/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated 

and will be removed in future. Use torchrun. 

Note that --use_env is set by default in torchrun. 

If your script expects `--local_rank` argument to be set, please 

change it to read from `os.environ['LOCAL_RANK']` instead. See 

https://pytorch.org/docs/stable/distributed.html#launch-utility for 

further instructions 

 

  FutureWarning, 

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 3875024) of binary: /root/miniconda3/envs/mmdetect_py1.11/bin/python3 

Traceback (most recent call last): 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/runpy.py", line 193, in _run_module_as_main 

    "__main__", mod_spec) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/runpy.py", line 85, in _run_code 

    exec(code, run_globals) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module> 

    main() 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main 

    launch(args) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch 

    run(args) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run 

    )(*cmd_args) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ 

    return launch_agent(self._config, self._entrypoint, list(args)) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent 

    failures=result.failures, 

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

======================================================== 

./tools/train.py FAILED 

-------------------------------------------------------- 

Failures: 

  <NO_OTHER_FAILURES> 

-------------------------------------------------------- 

Root Cause (first observed failure): 

[0]: 

  time      : 2023-08-02_02:33:57 

  host      : hw 

  rank      : 0 (local_rank: 0) 

  exitcode  : -6 (pid: 3875024) 

  error_file: <N/A> 

  traceback : Signal 6 (SIGABRT) received by PID 3875024 

======================================================== 

Process ForkServerProcess-2: 

Traceback (most recent call last): 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap 

    self.run() 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/process.py", line 99, in run 

    self._target(*self._args, **self._kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 61, in wrapper 

    raise exp 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 58, in wrapper 

    func(*args, **kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 275, in task_distribute 

    key, func_name, detail = resource_proxy[TASK_QUEUE].get() 

  File "<string>", line 2, in get 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod 

    kind, result = conn.recv() 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/connection.py", line 250, in recv 

    buf = self._recv_bytes() 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes 

    buf = self._recv(4) 

  File "/root/miniconda3/envs/mmdetect_py1.11/lib/python3.7/multiprocessing/connection.py", line 383, in _recv 

    raise EOFError 

EOFError