"bash:orted: command not found" Error

Symptom

In the cluster scenario, when the mpirun command is executed, the error message "bash: orted: command not found" is reported as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

bash: orted: command not found
--------------------------------------------------------------------------
A daemon (pid 8793) died unexpectedly with status 127 while attempting
to launch so we are aborting.
 
There may be more information reported by the environment (see above).
 
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.

Possible Cause

There are residual hccl_test processes in the cluster that have failed to exit.

Solution

Use MPI to terminate the residual hccl_test processes.

1. Prepare the Hostfile configured during HCCL Performance Tester execution, that is, the Hostfile defined in 4, for example, with the file name hostfile.
2. Terminate the participating hccl_test processes on all nodes in the cluster.
   • In the MPICH installation scenario, the example command is as follows:

     mpirun -f hostfile -n 512 pkill -9 -f "all_reduce_test|mpirun"

     • -f: MPICH command parameter, indicating the Hostfile node list file.
     • -n: MPICH command parameter, indicating the total number of NPUs to be terminated, that is, the number of nodes multiplied by the number of NPUs involved in training on each node. Modify it based on the actual situation.
     • pkill: Linux command. The subsequent -f is the pkill parameter, used to specify the pattern of the process name or command-line arguments to be matched. In the example command, all_reduce_test is the previously executed HCCL test command. Modify it based on the actual command executed.
   • In the Open MPI installation scenario, the example command is as follows:

     mpirun -hostfile hostfile -n 512 pkill -9 -f "all_reduce_test|openmpi"

     • -hostfile: Open MPI command parameter, indicating the Hostfile node list file.
     • -n: Open MPI command parameter, indicating the total number of NPUs to be terminated, that is, the number of nodes multiplied by the number of NPUs involved in training on each node. Modify it based on the actual situation.
     • pkill: Linux command. The subsequent -f is the pkill parameter, used to specify the pattern of the process name or command-line arguments to be matched. In the example command, all_reduce_test is the previously executed HCCL test command. Modify it based on the actual command executed.
3. After the preceding steps are completed, execute the HCCL Performance Tester again to perform the test.