Typical Operator Behavior Analysis

This section uses the profiling data of the AllReduce operator in the Atlas 800T A2 two-node cluster as an example to describe how to map task orchestration of the communication operator to the profiling tasks. The following figure shows the complete AllReduce operator execution process on a rank. It also maps the AllReduce operator execution steps to the profiling.

1. Copy the communication data from the user input memory to the HCCL buffer memory.

   

2. Implement the ReduceScatter communication semantics within a rank, including pre-notify synchronization, ReduceInline memory copy, on-the-fly calculation, and post-notify synchronization.

   

3. Implement the AllReduce communication semantics between ranks. Notify synchronization and data communication between ranks are implemented through RoCE, and Notify Record and data communication tasks are implemented by dispatching WQE using RDMASend. Therefore, during profiling, the combination of RDMASend (Notify Record) + Notify Wait corresponds to the inter-rank pre-notify and post-notify synchronization tasks, and the combination of RDMASend (data communication) + RDMASend (Notify Record) + Notify Wait corresponds to inter-rank data communication.

   

   In addition, you can obtain the local end, remote end, data size, and bandwidth information of the RDMASend (data communication) task from the task details.

   

4. Implement the AllGather communication semantics within a rank, including pre-notify synchronization, memcpy, and post-notify synchronization.

   

5. Copy the communication data from the HCCL buffer to the user output memory.