Parameter Consistency Verification (EI0005)

Symptom

The printed logs contain the keyword "The arguments for collective communication are inconsistent between ranks," as shown below:

EI0005: 2024-04-24-06:32:27.781.599 The arguments for collective communication are inconsistent between ranks:parameter count, local end 16512, remote end 8320
        TraceBack (most recent call last):
        Transport init error. Reason: [Create] [DestLink]Create Dest error! createLink para:rank[5]-localUserrank[4]-localIpAddr[127.10.0.1], dst_rank[6]-remoteUserrank[7]-remote_ip_addr[127.10.0.1]
        Transport init error. Reason: [Create] [DestLink]Create Dest error! createLink para:rank[5]-localUserrank[4]-localIpAddr[127.10.0.1], dst_rank[4]-remoteUserrank[5]-remote_ip_addr[127.10.0.1]
        call hccl op:HcomAllReduce(HcomAllReduce) load task fail[FUNC:Distribute][FILE:hccl_task_info.cc] [LINE:329]
        [[{[node Ge0p3_0]}]]

Alternatively, the CANN logs contain the keyword "CMD information *** check fail," as shown below:

[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.640 [rank_consistentcy_checker.cc:429] [3743951][InitChannelStage][ParameterConflict]CMD information tag check fail. local[AllGather_127.10.0.1%enp_60000_0_1761379874757928], remote[AllReduce_127.10.0.1%enp_60000_0_1761379874757928]
[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.666 [rank_consistentcy_checker.cc:439] [3743951][InitChannelStage][ParameterConflict]CMD information cmdType check fail. local[6], remote[2]
[ERROR] HCCL(3743927,all_reduce_test):2025-10-25-16:11:16.831.679 [rank_consistentcy_checker.cc:439] [3743951][InitChannelStage][ParameterConflict]CMD information op check fail. local[255], remote[0]

Possible Cause

During parameter plane connection setup, parameter consistency between the two ends will be verified after the socket is set up. The verification scope includes the operator identifier (tag), operator type (cmdType), reduction type (op), data size (count), HCCL buffer size (cclbufferSize), and data type (dataType). You can determine the inconsistent data based on the error information. In the following example, the operator identifiers (tag) of the two ends are inconsistent. As a result, the consistency verification fails when the communication operator sets up a connection. The data in local and remote is inconsistent on both ends.

You can check the information about the ranks with inconsistent parameters in the "Transport init error! createLink para:" error log. For instance, run the grep -r "Transport init error! createLink para:" command. The following information is displayed:

[ERROR] HCCL(3215542,all_reduce_test):2025-11-20-18:18:03.114.306 [transport_manager.cc:886] [3215599][InitChannelStage][Timeout]Transport init error! createLink para:rank[2]-localUserrank[2]-localIpAddr[127.10.0.1/2], remoteRank[1]-remoteUserrank[1]-remoteIpAddr[127.10.0.1/1], machineType[1], linkMode[1], isUsedRdma[0], tag[AllReduce_127.10.0.1%enp_60000_0_1763633852475745

• localUserrank: local rank ID
• localIpAddr: IP address of the local rank
• remoteUserrank: remote rank ID
• remoteIpAddr: IP address of the remote rank
• tag: communication operator identifier

Solution

1. If the function is normal when SuperKernel is not enabled but initialization inconsistency occurs after SuperKernel is enabled, you are advised to remove the HCCL operator from the calibration range of SuperKernel. For details, see GE Graph Mode > GE Graph Mode Functions > Calibrating the SuperKernel Range in a Graph in PyTorch Graph Mode User Guide (TorchAir).
2. Check the root cause for inconsistent operators dispatched by the two ends with inconsistent parameters based on the error information.

   Note: Some values in the logs are enumerated values. cmdType indicates the operator type, and op indicates the reduction type. The following tables list the enumerated value mappings.

   Table 1 Operator type corresponding to the enumerated value of cmdType

   Enumerated Value of cmdType	Operator Type
   1	BroadCast
   2	AllReduce
   3	Reduce
   4	Send
   5	Receive
   6	AllGather
   7	ReduceScatter
   8	AlltoAllV
   9	AlltoAllVC
   10	AlltoAll
   11	Gather
   12	Scatter
   13	BatchSendRecv
   16	AllGatherV
   17	ReduceScatterV

   Table 2 Reduction type corresponding to the enumerated value of op

   Enumerated Value of op	Reduction Type
   0	SUM
   1	PROD
   2	MAX
   3	MIN
   255	Non-Reduce operator