[object Object][object Object]

The LLM sparse quantization tool provides three functions: sparsification, quantization, and compression.

• Sparsification: The model sparsification tool uses an algorithm to determine the importance of each element in the model weight to the precision result and sets the weight value that has little impact on the final precision to zero.
• Quantization: Both the weights and activations are quantized. The high-precision floating-point numbers are converted into 8-bit numbers, which directly reduces the size of the weights and brings performance benefits.
• Compression: The weight compression tool further encodes and compresses model weights using a compression algorithm to minimize the weight size and generate compressed weights and an index file.

[object Object]

Weight directory structure after sparsification and quantization:

• Quantized outputs: [object Object] (weight file) and [object Object] (weight description file).
• The other files in the directory are required for inference, and they vary slightly by model.

The following is a partial view of [object Object] after quantization:

Quantized MatMul weights now include [object Object], [object Object], [object Object], and [object Object]. [object Object] and [object Object] are used to quantize the activations. MatMul uses the quantized activations and weights for computation. [object Object] and [object Object] are used to dequantize the MatMul computation result.

Weight directory structure after compression:

Before compression, the weights are loaded and split across devices. The compression algorithm needs to be executed based on the split weights.

The following shows part of the quantized weight description file [object Object]:

Compared with quantization, the compressed MatMul weights include an index. The compression information is used to restore the weights.

Figure 1 Process of inference with quantized weights [object Object][object Object]

Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])

[object Object]undefined[object Object]

Before using the sparse quantization script, install the msModelSlim tool. For details about the installation procedure, see

[object Object]

The following uses Qwen3-8B as an example:

1. Use the following instructions to generate the W8A8S-quantized weights.

   [object Object]
   • The preceding command demonstrates the optimal parameter configuration for generating the W8A8S sparse quantization weights of Qwen3-8B. Different models require different parameter configurations. Check the [object Object] file of the model for more details.
   • After the weights are generated, copy the [object Object] file of the floating-point weights to the W8A8S-quantized weight path.

2. Run the following command to set the environment variable of the Python path where msModelSlim is located. [object Object] is the Python path in the compilation procedure for msModelSlim installation.

   [object Object]

3. Run the following command to compress the quantized weights to generate the W8A8SC-quantized weights:

   [object Object]

   The number of TPs is the number of parallel tensors, which must be the same as the number of parallel tensors during weight running.

[object Object]

The following uses Qwen3-8B as an example. You can run the following commands to perform a dialog test. The inference content is "What's deep learning?".