ERROR CQE Error (EI0013)
In HCCL, ERROR CQE indicates that the number of RoCE packet retransmissions exceeds the upper limit. If this error occurs, the cluster will be suspended, causing timeout. HCCL periodically polls the RoCE driver to obtain its events. You can call the HcclGetCommAsyncError API to check whether an ERROR CQE occurs.
Symptom
The printed logs contain error code EI0013 and the keyword "Error ROCE CQE," as shown below:
[PID: 3448331] 2025-12-04-21:59:08.232.310 Execution Error ROCE CQE(EI0013): An error CQE occurred during operator execution. Local information: server 127.0.0.1, device ID 0, device IP 127.10.0.1. Peer information: server 127.0.0.2, device ID 1, device IP 127.10.0.2. Possible Cause: 1. The network between two devices is abnormal. For example, the network port is intermittently disconnected.2. The peer process exits abnormally in advance. As a result, the local end cannot receive the response from the peer end. Solution: 1. Check whether the network devices between the two ends are abnormal.2. Check whether the peer process exits first. If yes, check the cause of the process exit.
The CANN logs contain the keyword "error cqe status," as shown below:
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.612 [hns_roce_lite.c:630]hns_roce_lite_handle_error_cqe(630) : error cqe status: 0x15
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.622 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000000): 0x00041580
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.627 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000001): 0x00000000
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.630 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000002): 0x00000000
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.634 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000003): 0x1500047c
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.637 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000004): 0x00000000
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.640 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000005): 0x00000000
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.644 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000006): 0x00000000
[ERROR] ROCE(2040034,alltoall_test):2025-09-15-08:38:12.776.647 [hns_roce_lite.c:747]dump_err_cqe(747) : CQ(0x10) CQE(0x5) INDEX(0x00000007): 0x00000000
[ERROR] HCCP(2040034,alltoall_test):2025-09-15-08:38:12.776.650 [ra_hdc_lite.c:794]tid:2040458,ra_hdc_lite_period_poll_cqe : [create][ra_hdc_period_poll]failed CQE status[12], wr[0]
[ERROR] HCCL(2040034,alltoall_test):2025-09-15-08:38:13.607.432 [heartbeat.cc:1229] [2040666][TaskExecStage][HeartbeatAbnormal][ROCE CQE ERROR]cqe error status[12], time:[2025-09-15 08:38:12.776654],localInfo{server[127.10.0.1],deviceId[127.10.0.1],deviceIp[127.11.0.1]}, remoteIP{server[127.10.0.2],deviceId[127.10.0.2],deviceIp[127.11.0.2]}
Possible Cause
ERROR CQE occurs when the local end does not receive an acknowledgment from the remote end within a specified period of time after sending a packet to the remote end. This error indicates issues like network channel interruption, remote end disconnection, or the poor connection status. Besides network problems, ERROR CQE can also occur when the process on the remote end crashes.
Solution
Locate the remote end of ERROR CQE based on the error information.
[ERROR] HCCL(2040034,alltoall_test):2025-09-15-08:38:13.607.432 [heartbeat.cc:1229] [2040666][TaskExecStage][HeartbeatAbnormal][ROCE CQE ERROR]cqe error status[12], time:[2025-09-15 08:38:12.776654],localInfo{server[127.10.0.1],deviceId[127.10.0.1],deviceIp[127.11.0.1]}, remoteIP{server[127.10.0.2],deviceId[127.10.0.2],deviceIp[127.11.0.2]}
In the information, localIP and remoteIP indicate the device IP addresses of the local and remote ends, respectively. Locate the compute node or log where the corresponding rank is located based on the hardware resource information.
- Check whether the network is normal. Use the hccn_tool to check whether there are intermittent disconnection records of the network port. The following information indicates that the network port is disconnected at 10:13:50 2025. If a collective communication operator is executed at this time, an ERROR CQE is generated. Further locate the cause of the intermittent disconnection of the network port.
$ hccn_tool -i 0 -link_stat -g [devid 0]current time : Tue Oct 28 21:46:46 2025 [devid 0]link up count : 2 [devid 0]link down count : 1 [devid 0]link change records : [devid 0] Sun Oct 5 10:13:51 2025 LINK UP [devid 0] Sun Oct 5 10:13:50 2025 LINK DOWN [devid 0] Sun Oct 5 10:13:35 2025 LINK UP
- Check whether the remote service process exits abnormally or enters the resource destruction workflow when an ERROR CQE occurs on the local end. Check the remote service logs or plogs to determine whether the remote process exits abnormally before the ERROR CQE occurs on the local end.
- If the values of HCCL_RDMA_TIMEOUT (retransmission timeout period) and HCCL_RDMA_RETRY_CNT (retransmission times) are small, ERROR CQE may occur when the link status is poor. Increase the values of the environment variables to solve the problem.
status[12] indicates that the RoCE packet retransmission times out. Other status codes are rare. If you encounter one, contact technical support.