Relationship Between Communication Operator Retry and Overall Network Performance
After the HCCL communication operator retry feature is enabled, the end-to-end (E2E) performance change of the entire network is closely related to the model sharding and deployment mode. This section describes the relationship between the retry feature and network performance.
Key Communicator
A key communicator is a communicator whose performance change will cause a great change in the E2E performance of the entire network. This means that the communicator is very important and is the performance bottleneck of the entire network.
Generally, there are multiple communicators on the entire network, and there is usually one key communicator. This section focuses on the key communicator for performance analysis.
The following figure shows an example.
In the preceding profiling, there are four communicators where communication actually occurs, which are Group_777, Group_1289, Group_257, and Group_9.
The BatchSendRecv operator executed in Group_1289 is introduced by PipelineParallel. Generally, asynchronous communication is used, which can be asynchronous with computing and does not occupy a large proportion of time. This communicator is not a key communicator.
Group_777 and Group_9 have few operator execution operations, thus having little impact on the global network. They are also not key communicators.
Therefore, Group_257 is the key communicator. If the performance of this communicator deteriorates, the E2E performance of the entire network is directly affected.
Relationship Between Network Performance Deterioration and the Key Communicator
- Focus 1: whether the retry feature is enabled for the key communicator
In some common deployment modes, for example, tensor parallelism (TP) and data parallelism (DP) are used together, TP is the key communicator. If the TP range is within the server (TP ≤ 16), the E2E performance is not affected because the communication operator retry feature is not enabled on the server.
Non-key communicators have little impact on the performance of the entire network. The following table lists the data of the test model in the lab.
Model
Sharding Mode
Deterioration Ratio
Description
Llama3-8B
(running on a 64-die cluster)
TP = 16 (key communicator)
DP = 4
0.03%
The retry feature is enabled only for the DP of non-key communicators, which has little impact on the E2E performance.
GPT4_dropLess
(running on a 128-die cluster)
TP = 8 (key communicator)
PP = 1
EP = 1
CP = 16
0.99%
The retry feature is enabled only for context parallelism (CP) of non-key communicators, which has little impact on the E2E performance.
Qwen3-moe-235B (running on a 128-die cluster)
TP = 8 (key communicator)
PP = 1
EP = 64
-0.1%
The retry feature is enabled only for expert parallelism (EP) of non-key communicators, which has little impact on the E2E performance.
- Focus 2: whether the communication expansion and computation of the key communicator can overlap
If the retry feature is enabled for the key communicator, the performance of the communicator will definitely deteriorate. However, whether the deterioration causes the deterioration of the entire network depends on whether the AI CPU expansion of the key communicator can overlap with the computation.
After the retry feature is enabled for a single communicator, the primary change lies in switching from asynchronous operator expansion mode to synchronous operator expansion mode, as shown in the figure below. Specifically, the execution mode transitions from the upper scheme to the lower scheme.
Whether the communication expansion time can be overlapped with the computation time is the key factor that determines whether the communicator affects the E2E performance. The analysis needs to be performed based on the computation operator (model structure).
As shown in the following figure, the compute operator latency is only 50 μs. The gap between the preceding and succeeding communication operators under the AI_CPU expansion mode is 150 μs. Therefore, the overhead introduced by the retry feature is 100 μs (150 – 50). If this overhead occurs on the key communicator, it leads to E2E performance degradation.
However, the deterioration degree depends on the proportion of operators in the key communicator on the entire network (closely related to the model structure and deployment mode) and whether the expansion in this dimension can overlap with the computation.
For example, for the same EP64 sharding, different models have different deterioration effects.
Model
Sharding Mode
Deterioration Ratio
Description
DeepSeek V3 (running on a 64-die cluster)
EP = 64
0.06%
The retry feature is enabled for EPs within the key communicator. However, model compute latency is relatively long, and the overhead introduced by retry execution can be masked by computation. As a result, the E2E performance degradation of the entire network remains minor.
qwen3-moe-30b
(running on a 64-die cluster)
EP = 64
3%
The retry feature is enabled for EPs within the key communicator. The overhead introduced by retry execution cannot be masked by computation. As a result, the E2E performance of the entire network suffers obvious degradation.
Note: EPs within the key communicator are non-affine in cross-supernode scenarios. Enabling the retry feature further degrades the overall network performance.
Therefore, factors affecting the E2E performance of a model are closely related to the model structure. The impact of the retry feature on the performance of the entire network needs to be evaluated based on the actual situation.