Communication Operator Dispatch

Communication operators are dispatched at the CANN layer in the profiling data. As shown in the following figure, one AscendCL@hcom_allReduce_ record corresponds to one AllReduce operator dispatch.

Collective communication operators are orchestrated and dispatched on the host and asynchronously executed on the device. Generally, the dispatch time and asynchronous execution time of communication operators overlap with each other, which can fully utilize the device resources. When the dispatch of communication operators becomes a bottleneck, the device needs to wait for the dispatch of communication operators, causing bubbles. To improve the utilization, you need to optimize the dispatch performance of collective communication operators. The common optimization methods are as follows:

• The dispatch of communication operators is performed on the host. Therefore, the time required for dispatch is affected by the CPU scheduling on the host. You can use CPU core binding to prevent performance loss caused by CPU core switching, thus improving the dispatch performance of communication operators.
• Run the export HCCL_OP_EXPANSION_MODE="AIV" command to switch to the AIV mode. However, the AIV mode is only supported in limited scenarios. If multiple communicators are executed concurrently in the service, unexpected behaviors such as deadlock may occur due to core preemption.