AllToAllVOperation

Description

Sends data (the data volume can be customized) to all communication cards in the communicator and receives data from all communication cards.

Operator Context

Figure 1 Position of AllToAll in communication INT8 quantization

Input the quantized tensor X of int8 and send it to each node through AlltoAll. (In this example, int8 is used for communication, which improves the communication speed.)

Then, the sum operation of reduce is used to sum up x, the int8 data is dequantized to the float16 data, and AllGather is used for communication to transmit the calculation result to each node.

Operator Function Implementation

Sends data to all communication cards in the communicator (the data volume can be customized using parameters) and receives data from all communication cards (the data volume can be customized using parameters). Currently, only the Atlas A2 inference products is supported.

• Example
  Figure 2 Operator input and output example
  
• Python calculation process

  golden_result = []
  for j in range(rankSize):
      tensor_list = intensors[j].reshape(-1).tolist() # intensors[j] stands for intesor on the jth card.
      golden_result += tensor_list[sdispls[rank]:sdispls[rank]+sendCounts[rank]]
  size = sum(recvCounts)
  golden_out_tensor = torch.tensor(golden_result).reshape(1,size)

Application Scenarios

Send data to each node in many-to-many mode. It is an extension of AllGather, but different nodes collect different data from a node.

Figure 3 Comparison between AllToAll and AllGather

Applies to model parallelism, matrix transposition in model parallelism, and matrix transposition from data parallelism to model parallelism.

Definition

struct AllToAllVParam {
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::vector<int64_t> sendCounts;
    std::vector<int64_t> sdispls;
    std::vector<int64_t> recvCounts;
    std::vector<int64_t> rdispls;
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    uint8_t rsv[64] = {0};
};

Parameters

Input

Output

Restrictions

• The length of the array parameters sendCounts, recvCounts, sdispls and rdispls is equal to the value of ranksize, and the values of all elements in the array are greater than 0.
• Currently, only the Atlas A2 inference products is supported.
• The input tensor dimensions of each communication card are the same.
• The sum of sendCounts and recvCounts array elements cannot overflow int64, and the sum of recvCounts arrays must be greater than 0. For any element in rdispls, recvCounts[i] + rdispls[i] cannot be greater than recvCountsSum (the last dimension of the output). For any element in sdispls, sendCounts[i] + sdispls[i] cannot be greater than the data volume of the input tensor. For example, if the input tensor has shape [3, 4, 5] and the data amount is 60 (3 x 4 x 5), the input tensor will be considered as a tensor with shape [60] for AllToAllV computation.
• rank, rankSize, and rankRoot must meet the following conditions:
  • 0 ≤ rank < rankSize
  • 0 ≤ rankRoot < rankSize

• If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
• When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.

  rm -rf /dev/shm/sem.lccl*
  rm -rf /dev/shm/sem.hccl*
  ipcrm -a