Execution

Prerequisites

The firewall has been disabled in the operating environment.
The number of concurrent connections that can be processed by the master node is restricted by the Linux kernel parameters somaxconn and tcp_max_syn_backlog. Therefore, for large-scale cluster networking, setting small values for somaxconn and tcp_max_syn_backlog may cause some clients to exit abnormally, thus leading to cluster initialization failures.
In the scenario of large-scale cluster networking, you are advised to adjust the values of somaxconn and tcp_max_syn_backlog on the master node based on the number of clusters, for example:
```
sysctl -w net.core.somaxconn=65535 
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
```

Procedure

Check whether the rootinfo file exists.
Before running HCCL Performance Tester, check whether the /etc/hccl_rootInfo.json file exists. This file stores the entity ID (EID) for NPU communication and is automatically generated after the environment is deployed. If the file does not exist, click here to contact technical support.

Configure the environment variables on which the HCCL Performance Tester depends.

In the MPICH installation scenario:

          
               export INSTALL_DIR=/usr/local/Ascend/cann
export PATH=/usr/local/mpich/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/mpich/lib:${INSTALL_DIR}/lib64:$LD_LIBRARY_PATH

In the Open MPI installation scenario:

          
               export INSTALL_DIR=/usr/local/Ascend/cann
export PATH=/usr/local/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/openmpi/lib:${INSTALL_DIR}/lib64:$LD_LIBRARY_PATH

INSTALL_DIR is the CANN directory, and /usr/local/Ascend is the default installation path of the root user. If a common user is used or the installation path is specified, replace it with an actual path.

/usr/local/mpich and /usr/local/openmpi are the MPI installation paths. Replace them with actual paths.

There is no need to configure the preceding environment variables if they already exist.

Configure HCCL environment variables.

Configure the communication NIC information used during communicator initialization on the node where the training process is started.

Configure the NIC name used on the host side during HCCL initialization and the IP version used by the communication NIC. HCCL can obtain the host IP based on the configured NIC name to create a communicator.

           
                # IP version used by the communication NIC during HCCL initialization. AF_INET indicates that IPv4 is used. AF_INET6 indicates that IPv6 is used.
export HCCL_SOCKET_FAMILY=AF_INET

# The following formats of NIC names are supported. (Select one from the four formats. If multiple NICs are configured via the environment variables, separate them with commas (,). The first matched NIC is used as the communication NIC.)
# Exact match of the NIC
export HCCL_SOCKET_IFNAME==eth0,enp0   # Use the specified eth0 or enp0 NIC.
export HCCL_SOCKET_IFNAME=^=eth0,enp0     # Do not use the eth0 or enp0 NIC.

# Fuzzy match of the NIC
export HCCL_SOCKET_IFNAME=eth,enp       # Use all NIC prefixed with eth or enp.
export HCCL_SOCKET_IFNAME=^eth,enp      # Do not use any NIC prefixed with eth or enp.

Note: If the NIC names of different nodes participating in collective communication are different (for example, the NIC name of node 1 is eth1 and that of node 2 is eth2), and the environment variables will be synchronized from the current node to other nodes, configure environment variables using fuzzy match.

Adjust the timeout for socket connection establishment.
In the collective communication scenario, the default timeout for socket connection establishment between devices is 120s. However, when the master node needs to establish many connections and process a large amount of data, the default value cannot meet the requirements and needs to be adjusted.

For example, if the number of NICs in a cluster is 3,000, change the timeout to 240s. If the number of NICs in a cluster is 5,000, change the timeout to 600s.
```
export HCCL_CONNECT_TIMEOUT=600
```
Adjust the size of the shared buffer between NPUs.
In collective communication, each communicator occupies a 200 MB buffer by default. The buffer size can be adjusted using the environment variable HCCL_BUFFSIZE, in MB.

If there are many communicators in the cluster, the overall buffer usage increases, which may affect the normal storage of model data. In this case, you can decrease the value of this environment variable to reduce the buffer space occupied by the communicator. If the service model data volume is small but the communication data volume is large, you can increase the value of this environment variable to expand the buffer space occupied by the communicator, thereby improving data communication efficiency.

When the hccl_test tool is used to perform a performance test, the communication data size is large. In this case, you can increase the value of HCCL_BUFFSIZE to improve the data communication efficiency. For the collective communication operator, when the test data size exceeds the value of HCCL_BUFFSIZE, the performance may deteriorate. It is recommended that the value of HCCL_BUFFSIZE be greater than the test data size.

Example:
```
export HCCL_BUFFSIZE=2048
```
For more environment variables, see "Collective Communication" in Environment Variables.
(Optional) Configure the auxiliary environment variables of the HCCL Performance Tester.
- Specify the device to execute HCCL Performance Tester.
```
# The numbers following HCCL_TEST_USE_DEVS represent the IDs of the device where HCCL Performance Tester is to be executed. Use commas (,) to separate multiple device IDs.
export HCCL_TEST_USE_DEVS="4,5,6,7"
```
- Collect profile data during HCCL Performance Tester execution.
```
# 1 indicates that profiling is enabled, and 0 indicates that profiling is disabled. The default value is 0. If profiling is enabled, profiling data is collected when HCCL Performance Tester is executed.
export HCCL_TEST_PROFILING=1
# Specify the path for storing profiling data. The default path is /var/log/npu/profiling.
export HCCL_TEST_PROFILING_PATH=/home/profiling
```
  If HCCL_TEST_PROFILING is enabled, profiling data is generated in the directory specified by HCCL_TEST_PROFILING_PATH after HCCL Performance Tester is executed. For details about how to parse the profiling data, see Using the msprof Command to Parse, Query, and Export the Profile Data in Performance Tuning Tool.
  
  Enabling profiling affects the performance of the collective communication operator.

Configure the hostfile file.
The Hostfile file is used to specify the nodes on which the communication process needs to be started. It is a text file that needs to be customized.

The default Hostfile template named hostfile already exists in the ${INSTALL_DIR}/tools/hccl_test/ directory. You can edit the template or customize the file storage path and name.
- When MPICH is installed, only the IPv4 communication is supported. The content format is as follows:
```
{Node IP address}:{Number of processes on each node}
```
  For example, define the hostfile file name as hostfile. The content is as follows:
```
10.10.130.22:8
10.10.130.21:8
```
- When Open MPI is installed, both IPv4 and IPv6 communications are supported. The content format is as follows:
```
{Node name slots}={Number of processes on each node}
```
  For example, define the hostfile file name as hostfile. The content is as follows:
```
node3 slots=8
node4 slots=8
```
- For Atlas A3 training product / Atlas A3 inference product , in the Hostfile, you need to configure the information about the AI servers that belong to the same SuperPoD together. Assuming that there are two SuperPoDs whose IDs are 0 and 1, you need to configure the AI server information in SuperPoD 0 and then configure the AI server information in SuperPoD 1. The cross configurations of the AI server information in SuperPoDs 0 and 1 are not supported.
- There is no need to configure the hostfile file for single-server scenarios.
Execute the HCCL Performance Tester.
Execute the HCCL Performance Tester in the ${INSTALL_DIR}/tools/hccl_test directory.
- In the MPICH installation scenario, the command format is as follows:
```
mpirun [-f <hostfile>] -n <number> ./bin/<executable_file> [-p <npus>] [-b <minbytes>] [-e <maxbytes>] [-f <incfactor>] [-o <operator>] [-r <root>] [-d <datatype>] [-z <0/1>] [-n <iters_count>] [-w <warmup_iters_count>] [-c <0/1>]
```
  Command example:
```
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum
```
  - mpirun is followed by MPI options.
  - ./bin/<executable_file> is followed by the options of the HCCL Performance Tester.
  For details about the options of the MPICH and collective communication test commands, see Description of Command-Line Options. Note that the MPICH parameters provided in this document are only commonly used parameters. For details about how to use the MPICH parameters and how to resolve issues that may occur in practice, see the official MPICH documentation.
  In scenarios where MPICH is installed, executing the mpirun command synchronizes the environment variables of the current node to other nodes by default. If environment variable discrepancies exist across nodes, write the environment variable configuration commands into an execution script and pass that script to the mpirun command. For example:
  1. Create an execution script named run.sh and write the environment variables to be configured. The following is a content example:
    export HCCL_TEST_USE_DEVS="4,5,6,7" $1
  2. Run the mpirun command, for example:
    mpirun -n 4 ./run.sh "./all_reduce_test -b 8K -e 64M -f 2 -p 4"
- In the Open MPI installation scenario, the command format is as follows:
```
mpirun [--prefix <mpi_install_path>] [-hostfile <hostfile>] -n <number> -x <env> [--allow-run-as-root] [--mca <key value>] ./bin/<executable_file> [-p <npus>] [-b <minbytes>] [-e <maxbytes>] [-f <incfactor>] [-o <operator>] [-r <root>] [-d <datatype>] [-z <0/1>] [-n <iters_count>] [-w <warmup_iters_count>] [-c <0/1>]
```
  Command example:
```
mpirun --prefix /usr/local/openmpi -hostfile hostfile -x LD_LIBRARY_PATH -x HCCL_SOCKET_FAMILY -x HCCL_SOCKET_IFNAME -x HCCL_CONNECT_TIMEOUT -x HCCL_BUFFSIZE --allow-run-as-root --mca btl_tcp_if_include eth0 --mca opal_set_max_sys_limits 1 -n 16 ./bin/all_reduce_test -p 16 -b 8K -e 64M -i 0 -o sum -d fp32 -w 3 -n 3
```
  - mpirun is followed by MPI options.
  - ./bin/<executable_file> is followed by the options of the HCCL Performance Tester.
  For details about the options of the Open MPI and collective communication test commands, see Description of Command-Line Options. Note that the Open MPI parameters provided in this document are only commonly used parameters. For details about how to use the Open MPI parameters and how to resolve issues that may occur in practice, see the official Open MPI documentation.