Communication Operator Execution
The execution of communication operators corresponds to the Communication (HCCL) layer in the profile data, as shown in the following figure.
- Group: a communicator.
- Plane0-X: different communication streams. Each plane corresponds to a communication stream. HCCL communication operator orchestration fully utilizes HCCS physical link resources through multi-stream concurrency.
- hcom_allReduce_xx: communication operator execution process. You can view the time consumption, data size, and data type of the communication operator in the detailed information.
A communication operator consists of multiple orchestrated notify synchronization tasks and memcpy tasks. To display the communication task orchestration information in Profiling, you need to collect at least level-1 profiling data.
Synchronization Task
- Notify Record: synchronization task. Set the notify register to 1.
- Notify Wait: synchronization task. Wait until the notify register is set to 1, and then clear it.
- RDMASend: inter-device RoCE synchronization task. Set the notify register of the remote end to 1.
For a synchronization task, you can obtain the task duration, notify ID, local end (src rank), and remote end (dst rank) from the task details.
Data Communication Task
- Memcpy: memory copy task, which is used for intra-device or on-chip memory copy.
- Reduce_Inline: memory copy task, which is used to complete the associated reduction calculation during data copy.
- RDMASend: inter-device RoCE communication task, which corresponds to the inter-device memory copy task.
For a data communication task, you can obtain the task duration, local end (src rank), remote end (dst rank), data size (size), and bandwidth (bandwidth) from the task details.
- In the profiling data, the RDMASend task corresponds to a synchronization or data communication task. You can determine whether the task is a synchronization or data communication task based on its data size. The data size of a synchronization task is fixed at 4 bytes, and the data size of a data communication task is subject to the actual communication volume.
- If the RDMASend task is a data communication task, the task execution duration is not equal to the actual communication duration. The task execution duration is only the duration for dispatching the WQE of the communication task to the QP. The actual communication duration can be calculated based on the data size and bandwidth or refer to the duration of the next Notify Wait task.