服务化启动成功但hccl有报错
收藏回复举报
服务化启动成功但hccl有报错
t('forum.solved') 已解决
发表于2025-03-14 13:59:46
0 查看
[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32.637.331 [hccl_socketmanager.cc:628] [14154]the connection failure between this device and target device may 

be due to the following reasons 

[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637. 333 [hccl_socket_manager.cc:629][14154]1. the connectionbetween this device and the target device is ab 

norma1 

[ERROR] 

ERRoR HCCL168 ,minde_I_backend _connector225-2-9-237hg hcdelinesf hcom on this devi 

ce and the target device exceeds the timeout threshold. make sure this by keyworld [Entry- 

[ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32. 637.342 [hccl_socket_manager. cc:634] [14154]4. thebehavior of executing the calculation graph on this device 

and the target device is inconsistent 

[ERROR] HCCL{10684 ,mindie_1lm_backend_connector):2025-02-09-21: 04:32. 637.345 [hccl_socket _manager.cc:635][14154]5. The TLS switch isinconsistent, or the TLS certificate expires 

[ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04 :32. 637.347 [hccl_socketJmanager.cc:636][14154]6. If src and dstIP address can be pinged, check whether the dev 

ce IP address conflicts. 

[ERROR] HCCL(10684 ,mindie_11m_backend_connector):2025-02-09-21:04 :32. 637. 350 [hccl_socket_manager. cc:638] [14154]7. Now you can freely specify a port for 1istening and connecting 

If an invalid port is chosen, it may result in failed listeningand connection timeouts 

ERROR] HccL(10634 ,mindie_1Im_backend_connector):2025-02-09-21:04:32. 637.357 [transport_manager.cc:602][14154]ca]] trace: hcclRet -> 9 

ERROR] HCCL(10684 ,mindie_1Im_backend_connector):2025-02-09-21:04:32.637.363 [transport_manager. cc: 5131 14154]cal1 trace: hccTRet -> 

ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32. 637.462 [transport_manager.cc:235][12446İ [Create]errNo [Ox0000000005000006] transport create fail in thread, 

ocal rank[4j remote rank[12] 

[ERROR] HCCL(10684 ,mindie_1im_backend_connector):2025-02-09-21:04:32.747.212 [hccl_communicator. cc: 5074][12446] [A11ocAlgResource]A1loc transports failed, tag[Al1Reduce_hcc1_wor] 

C Lgr oupNHRA1TReduceMeshExecutor_host 

ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.221 [hccl_communicator. cc:3666] [12446Jcal] trace: hcclRet -> 6 

ERROR] HCCL(10684 ,mindie_1lm_backend_connector):2025-02-09-21:04:32.747.248 [hccl_communicator. cc:2485][12446]cal1 trace: hcclRet 

ERROR] HCCL(10684 ,mindie_1lmbackend_connector):2025-02-09-21:04:32.747.304 (hcc1_comm. cc:237][12446][Hcclcomm][A11Reduce_hcc1_wor1d_group]errNo[oxo000000000000006] index[o] 

ERROR] HCCL(10684,mindie_11mbackend_connector):2025-02-09-21:04:32.747.316 [op_base. cc:1202][i2446]cal1 trace:hcclRet 

٥ 

ERROR] HCcL(10684,mindie_1lm_backend_connector):2025-02-09-21:04:32.748.629 [reduce_scatter_nhr. cc:323][12446]errNo[Ox0000000005000002] ptr [linkLeft] is nu11ptr, returh ACCL 

PTR 

[ERRoR] нсcL(1об84 ,mindie_11m_bасkend_connector):2025-02-09-21: 04:32.748. б37 [reduce_scatter_nhr. cc:75][12446]cal1 trace: hcclRet -> 2 

ERROR HCCL(10684,mindie_1Im_backend_connector):2025-02-09-21:04:32.748.643 [a11_reduce_nhr. cc:47][12446][A1IReduceNHR][RunAsync] rank[O] count [1835008] failed in reducescater 

step

我要发帖子