Description

Sets the number of queue pairs (QPs) used for data transmission during RDMA communication between two ranks. By default, one QP is created.

This environment variable indicates the number of QPs required between two ranks. The value must be an integer. The value range is [1, 32], but [1, 8] is recommended. If the number of QPs exceeds 8, the performance gain cannot be ensured and the service may fail to run due to excessive memory usage. The default value is 1.

If HCCL_RDMA_QPS_PER_CONNECTION is set to N1, N1 QPs are created between every two ranks. The service data transmitted between the two ranks through RDMA is evenly distributed to the N1 QPs for parallel transmission.

After the multi-QP transmission function is enabled, you can use the environment variable HCCL_MULTI_QP_THRESHOLD to set the minimum threshold of the data size shared by each QP. If you want to specify the source port number used by each QP, you can use the environment variable HCCL_RDMA_QP_PORT_CONFIG_PATH.

Example

export HCCL_RDMA_QPS_PER_CONNECTION=4

Constraints

This environment variable supports only the single-operator calling mode and does not support the static graph mode.
The QP configuration priority is as follows:
Multi-QP configuration on the management plane (configured using the -s multi_qp parameter of hccn_tool) > QP configuration of the NSLB (configured using the -t nslb-dp parameter of hccn_tool) > Environment variable HCCL_RDMA_QP_PORT_CONFIG_PATH > Environment variable HCCL_RDMA_QPS_PER_CONNECTION

Applicability

Atlas A2 training product/Atlas A2 inference product (For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800T A2 training server, Atlas 900 A2 PoD cluster basic unit, and Atlas 200T A2 Box16 heterogeneous subrack are supported.)

Atlas A3 training product/Atlas A3 inference product