Parameter Consistency Verification (EI0005)

Symptom

The printed logs contain the keyword "The arguments for collective communication are inconsistent between ranks", as shown in the following:

EI0005: 2024-04-24-06:32:27.781.599 The arguments for collective communication are inconsistent between ranks:tag[HcomAllReduce_6629421139219749105_0], parameter[count], local[16512], remote [8320]
        Solution: Check whether the training script and ranktable of each NPU are consistent.
        TraceBack (most recent call last):
        Transport init error. Reason: [Create] [DestLink]Create Dest error! createLink para:rank[5]-localUserrank[4]-localIpAddr[127.10.0.1], dst_rank[6]-remoteUserrank[7]-remote_ip_addr[127.10.0.1]
        Transport init error. Reason: [Create] [DestLink]Create Dest error! createLink para:rank[5]-localUserrank[4]-localIpAddr[127.10.0.1], dst_rank[4]-remoteUserrank[5]-remote_ip_addr[127.10.0.1]
        call hccl op:HcomAllReduce(HcomAllReduce) load task fail[FUNC:Distribute][FILE:hccl_task_info.cc] [LINE:329]
        [[{[node Ge0p3_0]}]]

Alternatively, the CANN logs contain the keyword "CMD information *** check fail", as shown in the following:

[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.640 [rank_consistentcy_checker.cc:429] [3743951][InitChannelStage][ParameterConflict]CMD information tag check fail. local[AllGather_127.10.0.1%enp_60000_0_1761379874757928], remote[AllReduce_127.10.0.1%enp_60000_0_1761379874757928]
[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.666 [rank_consistentcy_checker.cc:439] [3743951][InitChannelStage][ParameterConflict]CMD information cmdType check fail. local[6], remote[2]
[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.679 [rank_consistentcy_checker.cc:439] [3743951][InitChannelStage][ParameterConflict]CMD information op check fail. local[255], remote[0]

Possible Cause

During parameter plane connection setup, parameter consistency between the two ends will be verified after the socket is set up. The verification scope includes the operator identifier (tag), operator type (cmdType), reduction type (op), data size (count), HCCL buffer size (cclbufferSize), and data type (dataType). You can determine the inconsistent data based on the error information. In the following example, the operator identifiers (tag) of the two ends are inconsistent. As a result, the consistency verification fails when the communication operator sets up a connection. The data in local and remote is inconsistent data on both ends.

You can determine the local and remote ranks with inconsistent parameters through the "Transport init error! createLink para:" error log. For example, you can run grep -r "Transport init error! createLink para:" to check the following result:

[ERROR] HCCL(3215542,all_reduce_test):2025-11-20-18:18:03.114.306 [transport_manager.cc:886] [3215599][InitChannelStage][Timeout]Transport init error! createLink para:rank[2]-localUserrank[2]-localIpAddr[127.10.0.1/2], remoteRank[1]-remoteUserrank[1]-remoteIpAddr[127.10.0.1/1], machineType[1], linkMode[1], isUsedRdma[0], tag[AllReduce_127.10.0.1%enp_60000_0_1763633852475745

• localUserrank: local rank ID
• localIpAddr: IP address of the local rank
• remoteUserrank: remote rank ID
• remoteIpAddr: IP address of the remote rank
• tag: communication operator identifier

Solution

Check the root cause for inconsistent operator delivery by the two ends with inconsistent parameters based on the error information.

Note: Some values in the logs are enumerated values. cmdType indicates the operator type, and op indicates the reduction type. The following tables list the enumerated value mappings.