[object Object][object Object]

Experts in Mixture-of-Experts (MoE) models refer to relatively independent feed-forward subnetworks, where a gating mechanism selects a subset of experts based on the input. MoE expert parallelism (EP) applies to MoE models with a substantial number of experts (e.g., dozens to hundreds). These models have a large total parameter count, yet the number of activated experts per iteration is far smaller than the total. In practice, MoE EP distributes numerous experts across multiple devices, enabling parallel execution of selected experts for inference or training. When deployed at scale, experts are sharded across multiple devices. Activating an expert triggers computation on its hosting device, along with necessary communication and scheduling during execution. MindIE Motor uses Prefill (P) and Decode (D) instances for distributed workload and relies on cluster management components (Coordinator/Controller) for scheduling and resource management. For details, see and .

You are advised to perform performance tuning according to the following process:

1. Analysis: Collect metrics such as latency, throughput, and NPU utilization to identify performance bottlenecks.
2. Locating: Determine whether the problem lies in the computing power, communication, memory, or P/D ratio.
3. Optimization: Adjust parameters such as the number of P/D instances and the number of nodes based on the typical configurations described below.
4. Verification: Retest key metrics and compare them with the baseline to confirm improvements.

[object Object]

Before performing MoE EP, ensure that the following conditions are met.

1. Supported hardware

   • For details about the hardware and OSs supported by the Atlas 800I A2 inference server, see . For details about PD disaggregation deployment and feature support, see and .
   • For details about the hardware and OSs supported by the Atlas 800I A3 SuperPoD server, see . For details about PD disaggregation deployment and feature support, see and .

2. Environment check

   • The environment has been checked before deployment and tuning. The specific check items are as follows:
   • Network topology: Node links, bandwidth, and latency meet deployment and collective communication requirements, with no packet loss or bit errors.
   • Storage performance: IOPS and bandwidth of the model load path meet the requirements for large model loading and execution.
   • NPU configuration: Model, on-chip memory, CANN/driver version, network port, and optical module specifications meet deployment requirements.
   • Supported OSs: For details, see .

3. Deployment preparation

   • Before tuning, check whether the PD disaggregation feature is supported by referring to and ensure that PD disaggregation has been deployed by referring to . PD disaggregation refers to the disaggregated deployment of Prefill and Decode instances to collaboratively complete inference.

[object Object]

The following table lists the typical configurations of the MoE EP solution. According to the current specifications, each Prefill compute instance (P instance) supports two compute nodes, and each Decode compute instance (D instance) supports four or eight compute nodes.
Each set of cluster management component instances (Coordinator and Controller instances) manage a set of PD instances and can manage up 96 nodes (24P + 6D). The following uses a cluster with 192 nodes as an example. The cluster needs to be divided into two independent sets of 24P + 6D instances, and two sets of cluster management component instances need to be created accordingly. The two sets of cluster management component instances can be deployed on a single node or on active and standby general-purpose compute nodes. If there are two general-purpose compute nodes hosting cluster management component instances onsite, you can also create primary/secondary cluster management component instances (primary/secondary Coordinator instances and primary/secondary Controller instances). General-purpose nodes are used to deploy cluster management components (Coordinator/Controller) and do not handle model inference or computation.

The PD instance scale managed by cluster management component instances can be flexibly set as required. For example, if there are 64 compute nodes, you may create a single set of cluster management component instances for all 64 nodes, or four sets of cluster management component instances corresponding to four independent PD instance sets (16 nodes per PD set).

Table 1 Typical configurations of the Atlas 800I A2 inference server

[object Object]undefined

[object Object]

Table 2 shows the typical hardware configuration of a general-purpose compute node. Currently, only the Arm architecture is supported for the CPU.

Table 2 Hardware requirements of the Atlas 800I A2 inference server general-purpose compute node

[object Object]undefined[object Object]

Table 3 lists the typical configuration of the MoE EP solution. According to the current specifications, each P instance uses one compute node, and each D instance can use one, two, four, or eight compute nodes.
Each set of cluster management component instances (Coordinator and Controller instances) manages an independent set of PD instances and one SuperPoD (48 compute nodes). If multiple SuperPoDs are deployed onsite and multiple independent sets of PD instances (48 nodes per set) are required, multiple sets of cluster management component instances need to be created accordingly. Multiple sets of cluster management component instances can be deployed on one or two general-purpose compute nodes. If there are two general-purpose compute nodes hosting cluster management component instances, you can also create primary/secondary cluster management component instances (primary/secondary Coordinator instances and primary/secondary Controller instances). General-purpose nodes are used to deploy cluster management components (Coordinator/Controller) and do not handle model inference or computation.

The PD instance scale managed by cluster management component instances can be flexibly set as required. For example, if there are 48 compute nodes, you may create a single set of cluster management component instances for all 48 nodes, or three sets of cluster management component instances corresponding to three independent PD instance sets (16 nodes per PD set).

Table 3 Typical configurations of the Atlas 800I A3 SuperPoD server

[object Object]undefined[object Object]

1. How do I verify that the performance is tuned after the configuration?

   • Business metrics: Use or your own stress testing tool under the same workload to compare Time to First Token (TTFT), output token throughput, end-to-end latency, etc., before and after tuning. The result should show improvements with no regression.
   • Data analysis: Use msProf or MindIE's msServiceProfiler to collect post‑tuning profile data (operator‑level and scheduler pipeline). Parse and visualize with MindStudio Insight to verify bottleneck relief and more balanced compute/communication ratios.
   • For details about how to use and configure the tools, see and .

2. What are the performance tuning methods in addition to parameter configuration?

   • Operator/graph: fusion and kernel selection, which depend on the inference engine and CANN.
   • Communication: topology, flow control, and collective communication algorithm and overlapping.
   • Running parameters: batch size, sequence length, and concurrency.
   • Quantization and precision policy: e.g., W8A8, applied under accuracy constraints.
   • Environment consistency: driver and CANN versions, firmware and BIOS settings, etc.
   • For targeted improvements, use profiling results from tools like msProf or msServiceProfiler, and re-validate using the aforementioned verification methods.