Overview
HCCL provides C and Python development APIs to implement distributed capabilities.
- C APIs implement framework adaptation in single-operator mode to achieve distributed capabilities.
- Python API implements framework adaptation in graph mode to achieve distributed optimization of TensorFlow networks on the AI processor.
This section describes how to call HCCL's C APIs to develop collective communication functions.
The following figure shows the process of calling HCCL's C APIs to implement collective communication functions.
Figure 1 Collective communication process
- Configure the cluster information, create a communicator handle, and initialize the HCCL communicator.
- Implement HCCL communications, including point-to-point communication and collective communication.
- Point-to-point communication involves directly transmitting data between two NPUs in a multi-NPU system. This method is typically used to transmit and receive activation values in pipeline parallel scenarios. HCCL provides point-to-point communication at different granularities, including single-rank RX and TX interfaces, as well as batch RX and TX interfaces.
- Collective communication means that multiple NPUs participate in data transmission operations, such as AllReduce, AllGather, and Broadcast. This method is typically used for gradient synchronization and parameter update across different NPUs in a large-scale cluster. Collective communication enables all compute nodes to exchange data in parallel, efficient, and orderly mode, improving data transmission efficiency.
- After collective communication is complete, destroy the communicator and free the memory and stream resources.