"retcode: 7" Is Returned During the HCCL Test

Symptom

In a cluster scenario, when the HCCL test command is executed, the HCCL Performance Tester starts successfully. However, after the table headers for the data size, time, and bandwidth are printed, the subsequent execution fails. The error log is as follows:

1
2
3
4
5
6
7
8
9

the minbytes is 8192, maxbytes is 2147483648, iters is 20, warmup_iters is 5
hccl interface return err ./common/src/hccl_test_common.cc:538, retcode: 7 
This is an error in init_hcclComm.
hccl interface return err ./common/src/hccl_test_common.cc:538, retcode: 7 
This is an error in init_hcclComm.
hccl interface return err ./common/src/hccl_test_common.cc:538, retcode: 7 
This is an error in init_hcclComm.
hccl interface return err ./common/src/hccl_test_common.cc:538, retcode: 7 
This is an error in init_hcclComm.

Possible Cause

Some hccl_test processes are still running on the node which is communicating with the current node in the cluster.

Solution

Use MPI to terminate the residual hccl_test processes.

1. Prepare the Hostfile configured during HCCL Performance Tester execution, that is, the Hostfile defined in 4, for example, with the file name hostfile.
2. Terminate the participating hccl_test processes on all nodes in the cluster.
   • In the MPICH installation scenario, the example command is as follows:

     mpirun -f hostfile -n 512 pkill -9 -f "all_reduce_test|mpirun"

     • -f: MPICH command parameter, indicating the Hostfile node list file.
     • -n: MPICH command parameter, indicating the total number of NPUs to be terminated, that is, the number of nodes multiplied by the number of NPUs involved in training on each node. Modify it based on the actual situation.
     • pkill: Linux command. The subsequent -f is the pkill parameter, used to specify the pattern of the process name or command-line arguments to be matched. In the example command, all_reduce_test is the previously executed HCCL test command. Modify it based on the actual command executed.
   • In the Open MPI installation scenario, the example command is as follows:

     mpirun -hostfile hostfile -n 512 pkill -9 -f "all_reduce_test|openmpi"

     • -hostfile: Open MPI command parameter, indicating the Hostfile node list file.
     • -n: Open MPI command parameter, indicating the total number of NPUs to be terminated, that is, the number of nodes multiplied by the number of NPUs involved in training on each node. Modify it based on the actual situation.
     • pkill: Linux command. The subsequent -f is the pkill parameter, used to specify the pattern of the process name or command-line arguments to be matched. In the example command, all_reduce_test is the previously executed HCCL test command. Modify it based on the actual command executed.
3. After the preceding steps are completed, execute the HCCL Performance Tester again to perform the test.