1）master pod报错信息
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:91 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 4
[ERROR] 2025-02-05-12:26:02 (PID:34, Device:0, RankID:0) ERR02200 DIST call hccl api failed.
EI0006: [PID: 34] 2025-02-05-12:26:02.963.740 Getting socket times out. Reason: Remote Rank did not send the data in time. Please check the reason for the rank being stuck
2）worker pod报错信息
TimeoutError: The client socket has timed out after 1800s while trying to connect to (xxxx,29501)
[ERROR] 2025-02-06-03:34:18 (PID:34, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
点赞
收藏
回复
分享
举报
已解决
1）master pod报错信息
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:91 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 4
[ERROR] 2025-02-05-12:26:02 (PID:34, Device:0, RankID:0) ERR02200 DIST call hccl api failed.
EI0006: [PID: 34] 2025-02-05-12:26:02.963.740 Getting socket times out. Reason: Remote Rank did not send the data in time. Please check the reason for the rank being stuck
2）worker pod报错信息
TimeoutError: The client socket has timed out after 1800s while trying to connect to (xxxx,29501)
[ERROR] 2025-02-06-03:34:18 (PID:34, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception