Basics

This section describes the theoretical background and development guide of MC² operators. Before learning this section, ensure that you are familiar with the knowledge of cube programming and HCCL collective communication database user guide.

MC² operators generally support the following product models:

Atlas A3 training products/Atlas A3 inference products

Atlas A2 training products/Atlas A2 inference products

MC² Operators

Compared with common compute or movement operators, MC² operators integrate serial communication and compute operations. By performing data tiling within the operator, the MC² operators enable parallel execution of compute and communication tasks, boosting operator performance. "MC² operators" are short for "Matrix Computation & Communication operators".

As shown in the following figure, the ideal execution duration of the serial communication operator and compute operator is the sum of the execution duration of the two operators. In the MC² operator fusing communication and compute tasks, required data for communication and compute is tiled to reduce the amount of data involved in each communication and compute task. By splitting overall communication and computing tasks into multiple batches, parallel execution of compute and communication is achieved, which greatly shortens the theoretical execution duration and delivers performance benefits.

Figure 1 Comparison of theoretical execution duration before and after communication-compute fusion

Application Scenarios and Advantages

As the model scale increases, training and inference on a single device encounter bottlenecks in compute capabilities, memory capacity, and energy efficiency. Therefore, distributed parallel compute becomes an inevitable technical path. Communication and compute tasks in the distributed training and inference of LLMs can be classified into two types based on the dependency between communication and compute.

The MC² technology is closely related to the network model structure. Generally, tightly coupled compute-communication tasks that meet the preceding conditions can potentially achieve performance improvement via MC² operators.