TLS Configuration Inconsistency (EI0016)

Symptom

The CANN logs contain the keyword "All ranks are consistent," as shown below:

[ERROR] HCCL(94774,all_reduce_test):2025-10-27-11:51:32.570.490 [topoinfo_exchange_agent.cc:831] [94774][InitGroupStage][RanktableCheck] Value Disable for config "tls" is invalid. Expected Value:"All ranks are consistent. Current status : rankList for enabled tls:[10.78.106.107/0]; rankList for disabled tls:[10.78.106.107/0]; rankList for query failure tls:".;

Possible Cause

During communicator creation, the server receives information about all ranks in the communicator, and checks whether TLS configurations of all these ranks are consistent. If the configurations are inconsistent, the verification fails. The system exits and prints a list of ranks with TLS disabled or enabled. Ranks not printed in the list have opposite TLS configurations.

This verification can be used only in Ascend HDK 25.2.0 or later in the scenario where the communicator is initialized through root information negotiation. The Atlas 350 Accelerator Card does not support this function.

Solution

1. Query the TLS switch status of each server involved in collective communication.

   Run the following command on each server to obtain the TLS switch status:

   hccn_tool -i <device_id> -tls -g

   <device_id> indicates the logical ID of the device. You can also run the following for statement to query the TLS information of all devices at once:

   for i in `seq 0 7`; do hccn_tool -i $i -tls -g; done    # 0 and 7 are the start and end values of the device ID to be queried.

   The command output is as follows:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

   dev_id:0, tls switch[0](0:disable, 1:enable), tls alarm time threshold[60]days dev_id:0, [pub cert] info: issuer[/C=CN/ST=GD/O=HUAWEI/OU=2012/CN=2_1thCA] start_time[Wed Feb 19 03:19:21 2020 GMT] end_time[Sat Feb 16 03:19:21 2030 GMT] dev_id:0, [ca1 cert] info: issuer[/C=CN/ST=GD/L=SZ/O=HUAWEI/CN=1thCA] start_time[Wed Feb 19 03:19:07 2020 GMT] end_time[Sat Feb 16 03:19:07 2030 GMT] dev_id:0, [ca2 cert] info: issuer[/C=CN/ST=GD/L=SZ/O=HUAWEI/CN=1thCA] start_time[Wed Feb 19 03:19:10 2020 GMT] end_time[Sat Feb 16 03:19:10 2030 GMT] dev_id:1, tls switch[0](0:disable, 1:enable), tls alarm time threshold[60]days dev_id:1, [pub cert] info: issuer[/C=CN/ST=GD/O=HUAWEI/OU=2012/CN=2_1thCA] start_time[Wed Feb 19 03:19:21 2020 GMT] end_time[Sat Feb 16 03:19:21 2030 GMT] dev_id:1, [ca1 cert] info: issuer[/C=CN/ST=GD/L=SZ/O=HUAWEI/CN=1thCA] start_time[Wed Feb 19 03:19:07 2020 GMT] end_time[Sat Feb 16 03:19:07 2030 GMT] dev_id:1, [ca2 cert] info: issuer[/C=CN/ST=GD/L=SZ/O=HUAWEI/CN=1thCA] start_time[Wed Feb 19 03:19:10 2020 GMT] end_time[Sat Feb 16 03:19:10 2030 GMT] ... ...

   tls switch[0] indicates that TLS is disabled, while tls switch[1] indicates that TLS is enabled.

2. Check whether the TLS states of all devices on each server are consistent.
   • If they are inconsistent, you are advised to toggle on all TLS switches. If TLS is disabled, information may be eavesdropped, tampered with, or forged during collective communication.

     You can run the following command to change the TLS status:

     hccn_tool -i <device_id> -tls -s enable 1

     enable indicates the switch to enable TLS. The value 1 indicates that TLS is enabled, and the value 0 indicates that TLS is disabled.

   • If they are consistent with TLS enabled, go to 3 to check whether the TLS certificate information on each rank is consistent.
3. Check whether the TLS certificate information of devices on all servers is consistent.

   You can check whether the TLS certificate information of the devices is consistent based on the information in 1. If the information is inconsistent, run the following command to replace the certificate suites:

   hccn_tool -i 0 -tls -s path /root pri pri.pem pub pub.pem ca1 ca1.pem ca2 ca2.pem crl xxx.crl

   -i indicates the device ID, -path indicates the path for storing the certificate, private key, or CRL, pri indicates the private key name, and pub indicates the device certificate file name. ca1, ca2, and crl indicate the root certificate file name, level-2 root certificate file name, and CRL file name, respectively.

   For more details about the usages and parameter descriptions of the hccn_tool, see the HCCN Tool API Reference of the corresponding version.