There are two key innovations in the traditional transformer structure for MoE. First, it replaces the Feed-Forward Network (FFN) with a Sparse MoE layer. Each FFN acts as an expert, but only a subset is activated per token during inference. The second innovation, the routing mechanism, is crucial for selecting the subset of experts to activate. The router determines which expert the token will enter at each layer. Thanks to the two mechanisms, MoE models can ensure an excellent model effect due to extensive expert knowledge. Compared with traditional models with the same number of parameters, MoE models guarantee high-performance inference by activating only some experts.
Typical models of the MoE structure include Mixtral 8x7B, Mixtral 8x22B, DeepSeek-16B-MoE, DeepSeek-V2, DeepSeek-V3, DeepSeek-R1, Qwen3-30B-A3B, and Qwen3-235B-A22B.
For details about the supported feature capabilities, see .
Table 1 Feature support matrix [object Object][object Object]
Model configuration parameters
For details about how to configure the inherent parameters of each model, see the [object Object] file in their official weight file.
The inference method for MoE models is identical to that of other models. You can follow the traditional LLM method during inference without setting any additional parameters.
The following uses DeepSeek-16B-MoE as an example. You can run the following commands to perform a dialog test. The inference content is "What's deep learning".