Feature

Re-execution

Re-execution can be enabled for MC² operators by configuring the compilation macro and environment variable, which prevents communication interruption caused by hardware intermittent disconnection in the environment where the communication task is executed. After re-execution is enabled for MC² operators, the AI CPU notifies the AI Core to re-deliver communication tasks using the mechanism shown in the following figure when detecting an environment exception, preventing communication interruption caused by hardware intermittent disconnection and improving communication stability.

Currently, the support for this capability is as follows:

Atlas A2 training products / Atlas A2 inference products does not support re-execution of MC² operators.

Atlas A3 training products / Atlas A3 inference products supports re-execution of MC² operators.

Figure 1 Re-execution mechanism of communication tasks

The re-execution conditions are as follows: