QuantCalibrationOp

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	√
Atlas training product	x

Note: For the products marked with x, no error is reported when the API is called, but no performance gains are obtained.

Description

Calibrates the output by calling the IFMR/HFMG quantization algorithm based on the user's quantization algorithm configuration during forward propagation, and outputs the quantization factors to the specified layer name in the corresponding record file record_file. This API is used for graph construction.

During forward propagation, the operator transparently transmits the source output, without modifying the activation input:

If the number of transferred data records is less than batch_num, the data accumulation method of the IFMR/HFMG operator is used to save the dataset.
If the number of transferred data records is equal to batch_num, the IFMR/HFMG algorithm is called to compute quantization factors. The format of the quantization factors is distinguished based on the value of quant_method, and the quantization factors are written to the corresponding record file. Currently, quant_method supports only kv_cache_quant.

Data is incrementally written into the record file. If data is overwritten, the system informs the user of the layers and corresponding parameters that are overwritten.

Prototype

QuantCalibrationOp (record_file,quant_algo_params, quant_method)

Parameters

Parameter	Input/Output	Description
record_file	Input	Path for storing the quantization factor record file. A string.
quant_algo_params	Input	Quantization algorithm and its quantization configuration. The format is as follows: { "act_algo": "hfmg", "num_bits": 8, "quant_granularity": "1", "with_offset": true, "batch_num": 1 } The configured fields vary according to the algorithm configured by act_algo. Table 1 describes the fields supported by the algorithm. A dict.
quant_method	Input	Quantization method. Defaults to kv_cache_quant, which specifies the format of the output quantization factors. A string.

Parameter

Input/Output

Description

record_file

Input

Path for storing the quantization factor record file.

A string.

quant_algo_params

Input

Quantization algorithm and its quantization configuration. The format is as follows:

{
  "act_algo": "hfmg",
  "num_bits": 8,
  "quant_granularity": "1",
  "with_offset": true,
  "batch_num": 1
}

The configured fields vary according to the algorithm configured by act_algo. Table 1 describes the fields supported by the algorithm.

A dict.

quant_method

Input

Quantization method. Defaults to kv_cache_quant, which specifies the format of the output quantization factors.

A string.

**Table 1** Fields supported by quant_algo_params
Field	Description	Fields Supported by IFMR	Fields Supported by HFMG
act_algo	Activation quantization algorithm. The options are as follows: ifmr (default): IFMR algorithm for activation quantization hfmg: HFMG algorithm for activation quantization	-	-
num_bits	Quantization bit width. Currently, the value can only be 8, indicating that the INT8 quantization bit width is used. This field is supported by both the IFMR and HFMG algorithms.	Yes	Yes
with_offset	Symmetric quantization or asymmetric quantization select for activation quantization. It is a global configuration parameter. true (default): asymmetric quantization false: symmetric quantization The asymmetric parameter takes precedence over the with_offset parameter if both of them exist in the configuration file. This field is supported by both the IFMR and HFMG algorithms.	Yes	Yes
batch_num	Batch number for quantization. The value must be greater than 0, and the default value is 1. This field is supported by both the IFMR and HFMG algorithms.	Yes	Yes
asymmetric	Symmetric or asymmetric quantization. It is used to select the layer-wise quantization algorithm. true (default): asymmetric quantization false: symmetric quantization This field is supported by both the IFMR and HFMG algorithms.	Yes	Yes
quant_granularity	Quantization granularity. The options are as follows: 0 (default): per_tensor. 1: per_channel. This field is supported by both the IFMR and HFMG algorithms.	Yes	Yes
max_percentile	Upper bound for searching for the largest of the IFMR activation quantization algorithm. The value range is (0.5, 1]. Defaults to 0.999999. This field is supported only by the IFMR algorithm.	Yes	No
min_percentile	Lower bound for searching for the smallest of the IFMR activation quantization algorithm. The value range is (0.5, 1]. Defaults to 0.999999. This field is supported only by the IFMR algorithm.	Yes	No
search_range	Quantization factor search range ([search_range_start, search_range_end]) of the IFMR algorithm. The value range is 0 < search_range_start < search_range_end. The recommended value range is [0.7, 1.3]. This field is supported only by the IFMR algorithm.	Yes	No
search_step	Quantization factor search step of the IFMR algorithm. The value range is (0, (search_range_end – search_range_start)]. Defaults to 0.01. This field is supported only by the IFMR algorithm.	Yes	No
num_of_bins	Number of bins (the minimum unit in a histogram). Value range: {1024, 2048, 4096, 8192}. Defaults to 4096. This field is supported only by the HFMG algorithm.	No	Yes

Returns

None

Example

import amct_pytorch as amct
from amct_pytorch.nn.module.quantization.quant_calibration_op import QuantCalibrationOp

class LinearNet(nn.Module):
    def __init__(self, quant_algo_params):

        super(LinearNet,self).__init__()
        self.quant_algo_params = quant_algo_params
        self.layer1 = nn.Linear(28, 1024, bias=False)
        self.ptq_1 = QuantCalibrationOp(record_file, quant_algo_params=self.quant_algo_params, quant_method="kv_cache_quant")

    def forward(self, layer_name, x):
        x = self.layer1(x)
        x = self.ptq_1(layer_name, x)

# Path for storing the quantization factor record file.
temp_folder = "./"
record_file = os.path.join(temp_folder, 'kv_cache.txt')
input_data = torch.randn((2, 2, 28, 28))

quant_algo_params = {"act_algo": "hfmg"}
model = LinearNet(quant_algo_params).to(torch.device("cpu"))
model.eval()

ans_2 = model("qat_1", input_data)

Parent topic: KV Cache Quantization APIs