quantize

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Description

Converts the high-precision model into a quantization calibration model and calculates the quantization parameters after inference.

Prototype

quantize(model, config)

Parameters

Parameter	Input/Output	Description
model	Input/Output	Input: high-precision model to be quantized Output: quantization calibration model Data type: torch.nn.Module
config	Input	Quantization configuration. Data type: user-defined dict, including the configurations of weight, input, algorithm, and skip_layers. For details about the configuration parameters, see config.

Parameter

Input/Output

Description

model

Input/Output

Input: high-precision model to be quantized

Output: quantization calibration model

Data type: torch.nn.Module

config

Input

Quantization configuration.

Data type: user-defined dict, including the configurations of weight, input, algorithm, and skip_layers. For details about the configuration parameters, see config.

Returns

None

Example

# Build a graph of the network for quantization.
ori_model = build_model()
model = copy.deepcopy(ori_model)
# Perform quantization configuration.
cfg = {
        'batch_num': 1,
        'quant_cfg': {
            'weights': {
                'type': 'int8',
                'symmetric': True,
                'strategy': 'tensor',
            },
        },
        'algorithm': {'minmax'},
        }
# Call the quantization API to generate a quantization calibration model.
quantize(model, cfg)

config

Key	-	-	Value
batch_num	-	-	Batch number for quantization. The value is of the uint32 type.
quant_cfg	-	-	Quantization configuration.
-	weights	-	Weight-only quantization configuration.
-	-	type	Weight quantization type. The value is of the string type. Currently, the following types are supported: hifloat8 float8_e4m3fn mxfp4_e2m1 float4_e2m1 int4 int8
-	-	symmetric	Whether the weight is symmetrically quantized. The value is of the boolean type. True: symmetric quantization False: asymmetric quantization
-	-	strategy	Weight quantization granularity. The value is of the string type. tensor: corresponding to per-tensor quantization channel: corresponding to per-channel quantization group: corresponding to per-group quantization. This parameter can be configured only when the quantized data type is MXFP8_E4M3FN. For details about the quantization granularity, see Compression Concepts.
-	-	group_size	Group size in per-group quantization. This parameter can be configured only after per-group quantization is configured. It is configured only in weight quantization scenarios. The value must fall within the range [32, K – 1] and must be a multiple of 32.
-	inputs	-	Activation quantization configuration.
-	-	type	Activation quantization type. The value is of the string type. Currently, the following types are supported: hifloat8 float8_e4m3fn mxfp8_e4m3fn int8
-	-	symmetric	Whether the activation is symmetrically quantized. The value is of the boolean type. True: symmetric quantization False: asymmetric quantization
-	-	strategy	Activation quantization granularity. The value is of the string type. tensor: corresponding to per-tensor quantization token: corresponding to per-token quantization
algorithm	-	-	Quantization algorithm. The value is of the string type. The following configurations are supported: awq: grids_num (uint32), number of search grids. Default: 20. gptq minmax smoothquant: smooth_strength (float), transfer strength. Default: 0.5. ofmr mxquant: MX data type conversion only. For details, see Compression Algorithms.
skip_layers	-	-	Layers to exclude from quantization. It is a global parameter of the string type. Any layer whose name contains the user-specified string is excluded.

Parent topic: Torch Module-based Quantization APIs