Feature List

MindIE LLM supports foundational, quantization, long-sequence, scheduling, acceleration, and interaction features. For details on enabling each feature and its limitations, refer to the links in the overview.

Category	Feature	Description	Benefits
Basic features	Multi-LoRA	Uses different LoRA weights for inference. For details, see Multi-LoRA.	Support the LoRA feature and dynamically loads and unloads weights.
	MoE	Enables sparse-activated expert networks to scale up model parameters without significantly increasing computational cost, thereby enhancing model capability. For details, see MoE.	Accommodates massive knowledge with trillions of parameters, outperforming dense models in potential performance.
	MLA	Uses low-rank key-value joint compression to eliminate inference bottlenecks and enable efficient inference. For details, see MLA.	Efficiently processes ultra-long contexts.
	Load balancing	Reduces the imbalance between NPUs, thereby improving the model inference performance. For details, see Load balancing.	Reduce delay.
	External shared expert	Deploys shared experts on a dedicated NPU card, separating them from routing and redundant experts. For details, see External shared expert.	Optimize TPOT.
	Expert parallel	Deploys experts across devices to enable expert-level parallel computation. For details, see Expert Parallel.	Reduce the latency and increase throughput.
	Data parallel	Batch splits inference requests across devices for parallel processing. For details, see Data Parallel.	Increase throughput.
	Tensor parallel	Shard tensors (e.g., weight matrices, activations) across multiple devices (e.g., NPUs) to enable distributed model inference. For details, see Tensor Parallel.	Reduce graphics memory per card.
Quantization features	Anti-outlier	Suppress outliers in data to improve model quantization accuracy. For details, see Anti-outlier.	Reduce quantization accuracy drop.
	PD MIX quantization	Uses different quantization modes in the prefill and decode phases of model inference. For details, see PD MIX Quantization.	Reduce graphics memory.
	W8A8 quantization	Quantizes weights and activations to the int8 format to reduce the model size and accelerate inference computation. For details, see W8A8 Quantization.	Reduce graphics memory and increase throughput.
	W4A8 mixed quantization	Quantizes model layers differentially: apply 4/8-bit hierarchical quantization to weights, and uniformly quantize activations to 8-bit. For details, see W4A8 Mixed Quantization.	Reduce graphics memory and increase throughput.
	W8A16 quantization	Quantizes weights to 8-bit only. For details, see W8A16 Quantization.	Reduce graphics memory and increase throughput.
	Attention quantization	Quantizes Q, K, and V to 8-bit, effectively compressing KV cache memory, accelerating attention computation during decoding, and significantly improving model throughput. For details, see Attention Quantization.	Reduce graphics memory and increase throughput.
	FA3 quantization	Quantize non-RoPE tensors of `k` to 8-bit using Attention-like quantization, while leaving RoPE tensors of `k` unquantized. This reduces KV cache memory usage and improves decoding speed, thereby increasing throughput. For details, see FA3 Quantization.	Reduce graphics memory and increase throughput.
	KV Cache Int8	Reduces memory usage and increases throughput by reducing KV memory usage and recomputation. For details, see KV Cache Int8.	Reduce graphics memory and increase throughput.
	W8A8SC sparse quantization	Accelerates the model by zeroing out unimportant weights via sparsification, converting high-precision values to low-bit-width storage, and further reducing the weight size using compression algorithms. For details, see W8A8SC Sparse Quantization.	Lower VRAM usage and increase maximum throughput via high sparsity.
	W16A16SC quantization	Sparsifies model weights via algorithm, then compresses and stores them using floating-point sparse quantization. For details, see W16A16SC Quantization.	Use high sparsity to increase throughput and avoid dequantization.
Long sequence features	Context parallel	Shards long sequences across the context dimension and distributes them to different devices for parallel processing, reducing Time To First Token (TTFT). For details, see Context Parallel.	Reduce memory usage and lower TTFT.
	Sequence parallel	Shards the KV cache so each card stores a distinct portion, reducing memory usage and enabling long-sequence support. For details, see Sequence Parallel.	Reduce graphics memory.
Scheduling features	Asynchronous scheduling	For scenarios with a large `maxBatchSize` and long input/output lengths, overlaps model inference with data preparation and result return, preventing wasted NPU compute and memory resources. For details, see Asynchronous Scheduling.	Reduce latency.
	SplitFuse	Breaks long prompts into smaller chunks and schedules them across multiple forward steps to reduce prefill latency. For details, see SplitFuse.	Reduce graphics memory and latency, and increase throughput
	SLO scheduling tuning	Improves the system throughput while ensuring the SLO. For details, see SLO Scheduling Tuning.	Increase throughput.
Acceleration features	Micro batch	Splits data into smaller batches during processing to fully utilize hardware resources and improve inference throughput. For details, see Micro Batch.	Increase throughput.
	Parallel decoding	Leverages compute capacity to mitigate memory bandwidth bottlenecks, improving compute utilization. For details, see Parallel Decoding.	Increase throughput.
	MTP	During inference, predicts multiple tokens at once rather than just the next one, significantly improving generation speed. For details, see MTP.	Increase throughput.
	Prefix cache	Reuses the KV cache corresponding to the repeated blocks across requests, reducing the prefill time. For details, see Prefix Cache.	Lower TTFT.
	KV cache pooling	Integrates larger storage media—such as DRAM and even SSDs—into the prefix cache pool to break through VRAM capacity limits. For details, see KV Cache Pooling.	Improve the prefix cache hit ratio.
Interaction	Function call	Supports function calls, enabling the foundation model to use tools. For details, see Function Call.	Enable the use of external tools to expand the application scope.
	Thinking analysis	Structurally parses the output of the foundation model and separates the thinking process from the output result. For details, see Thinking Analysis.	Improve the inference performance in complex scenarios.
	Thinking budget	Controls the depth of model thinking. When the thinking content exceeds the specified thinking_budget, the system uses a prompt to truncate the thinking process. For details, see Thinking Budget.	Improved inference performance in complex scenarios.
Others	Offline weight partitioning	Pre-loads shard weights into tmpfs to optimize large-scale model loading and reduce NPU transfer time. For details, see Offline Weight Partitioning.	Reduce the time required for loading weights.

Feature Combination Matrix

The compatibility of several features is indicated by the following symbols:

• ✅ = Fully compatible
• ❌ = Incompatible
• ❔ = To be determined

DeepSeek models

Qwen models