[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32.637.331 [hccl_socketmanager.cc:628] [14154]the connection failure between this device and target device may
be due to the following reasons
[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637. 333 [hccl_socket_manager.cc:629][14154]1. the connectionbetween this device and the target device is ab
norma1
[ERROR]
ERRoR HCCL168 ,minde_I_backend _connector225-2-9-237hg hcdelinesf hcom on this devi
ce and the target device exceeds the timeout threshold. make sure this by keyworld [Entry-
[ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32. 637.342 [hccl_socket_manager. cc:634] [14154]4. thebehavior of executing the calculation graph on this device
and the target device is inconsistent
[ERROR] HCCL{10684 ,mindie_1lm_backend_connector):2025-02-09-21: 04:32. 637.345 [hccl_socket _manager.cc:635][14154]5. The TLS switch isinconsistent, or the TLS certificate expires
[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637.347 [hccl_socketJmanager.cc:636][14154]6. If src and dstIP address can be pinged, check whether the dev
ce IP address conflicts.
[ERROR] HCCL(10684 ,mindie_11m_backend_connector):2025-02-09-21:04 :32. 637. 350 [hccl_socket_manager. cc:638] [14154]7. Now you can freely specify a port for 1istening and connecting
If an invalid port is chosen, it may result in failed listeningand connection timeouts
ERROR] HccL(10634 ,mindie_1Im_backend_connector):2025-02-09-21:04:32. 637.357 [transport_manager.cc:602][14154]ca]] trace: hcclRet -> 9
ERROR] HCCL(10684 ,mindie_1Im_backend_connector):2025-02-09-21:04:32.637.363 [transport_manager. cc: 5131 14154]cal1 trace: hccTRet ->
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32. 637.462 [transport_manager.cc:235][12446İ [Create]errNo [Ox0000000005000006] transport create fail in thread,
ocal rank[4j remote rank[12]
[ERROR] HCCL(10684 ,mindie_1im_backend_connector):2025-02-09-21:04:32.747.212 [hccl_communicator. cc: 5074][12446] [A11ocAlgResource]A1loc transports failed, tag[Al1Reduce_hcc1_wor]
C Lgr oupNHRA1TReduceMeshExecutor_host
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.221 [hccl_communicator. cc:3666] [12446Jcal] trace: hcclRet -> 6
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.248 [hccl_communicator. cc:2485][12446]cal1 trace: hcclRet
ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32.747.304 (hcc1_comm. cc:237][12446][Hcclcomm][A11Reduce_hcc1_wor1d_group]errNo[oxo000000000000006] index[o]
ERROR] HCCL(10684,mindie_11mbackend_connector):2025-02-09-21:04:32.747.316 [op_base. cc:1202][i2446]cal1 trace:hcclRet
٥
ERROR] HCcL(10684,mindie_1lm_backend_connector):2025-02-09-21:04:32.748.629 [reduce_scatter_nhr. cc:323][12446]errNo[Ox0000000005000002] ptr [linkLeft] is nu11ptr, returh ACCL
PTR
[ERRoR] нсcL(1об84 ,mindie_11m_bасkend_connector):2025-02-09-21: 04:32.748. б37 [reduce_scatter_nhr. cc:75][12446]cal1 trace: hcclRet -> 2
ERROR HCCL(10684,mindie_1Im_backend_connector):2025-02-09-21:04:32.748.643 [a11_reduce_nhr. cc:47][12446][A1IReduceNHR][RunAsync] rank[O] count [1835008] failed in reducescater
step
be due to the following reasons
[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637. 333 [hccl_socket_manager.cc:629][14154]1. the connectionbetween this device and the target device is ab
norma1
[ERROR]
ERRoR HCCL168 ,minde_I_backend _connector225-2-9-237hg hcdelinesf hcom on this devi
ce and the target device exceeds the timeout threshold. make sure this by keyworld [Entry-
[ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32. 637.342 [hccl_socket_manager. cc:634] [14154]4. thebehavior of executing the calculation graph on this device
and the target device is inconsistent
[ERROR] HCCL{10684 ,mindie_1lm_backend_connector):2025-02-09-21: 04:32. 637.345 [hccl_socket _manager.cc:635][14154]5. The TLS switch isinconsistent, or the TLS certificate expires
[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637.347 [hccl_socketJmanager.cc:636][14154]6. If src and dstIP address can be pinged, check whether the dev
ce IP address conflicts.
[ERROR] HCCL(10684 ,mindie_11m_backend_connector):2025-02-09-21:04 :32. 637. 350 [hccl_socket_manager. cc:638] [14154]7. Now you can freely specify a port for 1istening and connecting
If an invalid port is chosen, it may result in failed listeningand connection timeouts
ERROR] HccL(10634 ,mindie_1Im_backend_connector):2025-02-09-21:04:32. 637.357 [transport_manager.cc:602][14154]ca]] trace: hcclRet -> 9
ERROR] HCCL(10684 ,mindie_1Im_backend_connector):2025-02-09-21:04:32.637.363 [transport_manager. cc: 5131 14154]cal1 trace: hccTRet ->
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32. 637.462 [transport_manager.cc:235][12446İ [Create]errNo [Ox0000000005000006] transport create fail in thread,
ocal rank[4j remote rank[12]
[ERROR] HCCL(10684 ,mindie_1im_backend_connector):2025-02-09-21:04:32.747.212 [hccl_communicator. cc: 5074][12446] [A11ocAlgResource]A1loc transports failed, tag[Al1Reduce_hcc1_wor]
C Lgr oupNHRA1TReduceMeshExecutor_host
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.221 [hccl_communicator. cc:3666] [12446Jcal] trace: hcclRet -> 6
ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.248 [hccl_communicator. cc:2485][12446]cal1 trace: hcclRet
ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32.747.304 (hcc1_comm. cc:237][12446][Hcclcomm][A11Reduce_hcc1_wor1d_group]errNo[oxo000000000000006] index[o]
ERROR] HCCL(10684,mindie_11mbackend_connector):2025-02-09-21:04:32.747.316 [op_base. cc:1202][i2446]cal1 trace:hcclRet
٥
ERROR] HCcL(10684,mindie_1lm_backend_connector):2025-02-09-21:04:32.748.629 [reduce_scatter_nhr. cc:323][12446]errNo[Ox0000000005000002] ptr [linkLeft] is nu11ptr, returh ACCL
PTR
[ERRoR] нсcL(1об84 ,mindie_11m_bасkend_connector):2025-02-09-21: 04:32.748. б37 [reduce_scatter_nhr. cc:75][12446]cal1 trace: hcclRet -> 2
ERROR HCCL(10684,mindie_1Im_backend_connector):2025-02-09-21:04:32.748.643 [a11_reduce_nhr. cc:47][12446][A1IReduceNHR][RunAsync] rank[O] count [1835008] failed in reducescater
step