create_quant_cali_config

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	√
Atlas training product	x

Note: For the products marked with x, no error is reported when the API is called, but no performance gains are obtained.

Description

Generates the detailed quantization configuration (mainly about the KV cache) of each layer based on the model, quantization layer information, and quantization configuration input by users.

Prototype

create_quant_cali_config(config_file,model,quant_layers=None,config_defination=None)

Parameters

Parameter	Input/Output	Description
config_file	Input	Path (including the file name) of the quantization configuration file to be generated. The file is in JSON format and contains the quantization configuration of each KV cache quantization layer. The existing file (if any) in the path will be overwritten upon this API call. A string.
model	Input	Original model. Data type: torch.nn.Module
quant_layers	Input	Quantization layer information, a dictionary. If a simplified quantization configuration file is passed as an argument, the information depends on the configuration file. An example of KV cache quantization is as follows: {'kv_cache_quant_layers': ['MatMul_1']} Default: None A dict. Restrictions: quant_layers can be specified in a parameter (when set to None) or added to a simplified configuration file.
config_defination	Input	Simplified quantization configuration file. The simplified configuration file quant.cfg is generated based on quant_calibration_config_pytorch.proto. The .proto file is stored in /amct_pytorch/proto/* under the AMCT installation directory. For details about the parameters in the .proto file and the generated simplified quantization configuration file quant.cfg, see Simplified KV Cache Quantization Configuration File. Default: None* A string.

Returns

None

Example

import amct_pytorch as amct
# Build a graph of the network to be quantized.
model = build_model()
model.load_state_dict(torch.load(state_dict_path))
input_data = tuple([torch.randn(input_shape)])

# Create a quantization configuration file.
amct.create_quant_cali_config(config_file="./configs/config.json",
                             model=model,
                             quant_layers=None,
                             config_defination="./configs/quant.cfg")

Flush file:

A quantization configuration file in JSON format. (When quantization is performed again, the configuration file output by the API will be overwritten.) An example is as follows:

{
    "batch_num":1,
    "activation_offset":true,
    "matmul1":{
        "kv_data_quant_config":{
            "act_algo":"hfmg",
            "num_of_bins":4096,
            "quant_granularity":0
        }
    },
    "matmul2":{
        "kv_data_quant_config":{
            "act_algo":"hfmg",
            "num_of_bins":4096,
            "quant_granularity":0
        }
    },
    "matmul3":{
        "kv_data_quant_config":{
            "act_algo":"ifmr",
            "max_percentile":0.999999,
            "min_percentile":0.999999,
            "search_range":[
                0.7,
                1.3
            ],
            "search_step":0.01,
            "quant_granularity":0
        }
    }
}

Parent topic: KV Cache Quantization APIs