HCCL_RDMA_QPS_PER_CONNECTION
Description
Sets the number of queue pairs (QPs) used for data transmission during RDMA communication between two ranks. By default, one QP is created.
This environment variable indicates the number of QPs required between two ranks. The value must be an integer. The value range is [1, 32], but [1, 8] is recommended. If the number of QPs exceeds 8, the performance gain cannot be ensured and the service may fail to run due to excessive memory usage. The default value is 1.
If HCCL_RDMA_QPS_PER_CONNECTION is set to N1, N1 QPs are created between every two ranks. The service data transmitted between the two ranks through RDMA is evenly distributed to the N1 QPs for parallel transmission.
After the multi-QP transmission function is enabled, you can use the environment variable HCCL_MULTI_QP_THRESHOLD to set the minimum threshold of the data size shared by each QP. If you want to specify the source port number used by each QP, you can use the environment variable HCCL_RDMA_QP_PORT_CONFIG_PATH.
Example
export HCCL_RDMA_QPS_PER_CONNECTION=4
Restrictions
- This environment variable supports only the single-operator calling mode and does not support the static graph mode.
- The QP configuration priority is as follows:
Multi-QP configuration on the management plane (configured using the -s multi_qp parameter of hccn_tool) > QP configuration of the NSLB (configured using the -t nslb-dp parameter of hccn_tool) > Environment variable HCCL_RDMA_QP_PORT_CONFIG_PATH > Environment variable HCCL_RDMA_QPS_PER_CONNECTION