[object Object][object Object]

This quantization mode is floating-point sparse quantization. The following two operations are performed on the existing weights:

• Sparsification: The model sparsification tool uses an algorithm to determine the importance of each element in the model weights to the accuracy result and sets the weights that have little impact on the final accuracy to zero.

• Compression: The weight compression tool further encodes and compresses model weights using a compression algorithm to minimize the weight size and generate compressed weights and an index file.

  [object Object]

  Weight directory structure after sparsification:

  [object Object]
  • The sparse output includes: [object Object] (weight file) and [object Object] (weight description file).
  • The other files in the directory are required for inference, and they vary slightly by model.

  The following shows part of the sparsified weight description file [object Object]:

  [object Object]

  Weight directory structure after compression:

  [object Object]

  Before compression, the weights are loaded and split across devices. The compression algorithm must be executed based on the split weights.

  The following shows part of the quantized weight description file [object Object]:

  [object Object]

  Compared with sparsification, the compressed MatMul weights have [object Object] and [object Object] added. [object Object] is used to restore the weight, and [object Object] is used to restore the output dimension.

  Figure 1 Process of inference with quantized weights[object Object][object Object]

Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])

[object Object]undefined[object Object]

Before using the sparse quantization script, install the msModelSlim tool. For details about the installation procedure, see .

[object Object]

The following uses Qwen3-32B as an example:

1. Run the following command to generate the W16A16S floating-point sparse weights:

   [object Object]
   • The preceding command contains the optimal parameter configuration for generating the Qwen3-32B W16A16S floating-point sparse weights.
   • This quantization method has been integrated into the one-click quantization function of the msModelSlim tool. For details about the parameter configuration, see ".

2. Use the following instruction to set the environment variable of the Python path where msModelSlim is located. [object Object] is the Python path in the compilation procedure for msModelSlim installation.

   [object Object]

3. Run the following command to compress the floating-point sparse weights to generate the W16A16SC-quantized weights:

   [object Object]

   The number of TPs is the number of parallel tensors, which must be the same as the number of parallel tensors during weight inference.

[object Object]

The following uses Qwen3-32B as an example. You can run the following commands to perform a dialog test. The inference content is "What's deep learning?".