create_quant_cali_config

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	x
Atlas inference products	√
Atlas training products	x

Note: For the products marked with x, no error is reported when the API is called, but no performance gains are obtained.

Description

Generates the detailed quantization configuration (mainly about the KV cache) of each layer based on the model, quantization layer information, and quantization configuration input by users.

Prototype

create_quant_cali_config(config_file,model,quant_layers=None,config_defination=None)

Parameters

Parameter	Input/Output	Description
config_file	Input	Path (including the file name) of the quantization configuration file to be generated. The file is in JSON format and contains the quantization configuration of each KV cache quantization layer. The existing file (if any) in the path will be overwritten upon this API call. A string.
model	Input	Original model. A torch.nn.Module.
quant_layers	Input	Quantization layer information, a dictionary. If a simplified quantization configuration file is passed as an argument, the information depends on the configuration file. An example of KV cache quantization is as follows: {'kv_cache_quant_layers': ['MatMul_1']} Default: None A dict. Restrictions: quant_layers can be specified in a parameter (when set to None) or added to a simplified configuration file.
config_defination	Input	Simplified quantization configuration file. The simplified configuration file quant.cfg is generated based on quant_calibration_config_pytorch.proto. The .proto file is stored in /amct_pytorch/proto/* under the AMCT installation directory. For details about the parameters in the .proto file and the generated simplified quantization configuration file quant.cfg, see Simplified KV Cache Quantization Configuration File. Default: None* A string.

Returns

None

Example

import amct_pytorch as amct
# Build a graph of the network to be quantized.
model = build_model()
model.load_state_dict(torch.load(state_dict_path))
input_data = tuple([torch.randn(input_shape)])

# Create a quantization configuration file.
amct.create_quant_cali_config(config_file="./configs/config.json",
                             model=model,
                             quant_layers=None,
                             config_defination="./configs/quant.cfg")

Flush file:

A quantization configuration file in JSON format. (When quantization is performed again, the configuration file output by the API will be overwritten.) An example is as follows:

{
    "batch_num":1,
    "activation_offset":true,
    "matmul1":{
        "kv_data_quant_config":{
            "act_algo":"hfmg",
            "num_of_bins":4096,
            "quant_granularity":0
        }
    },
    "matmul2":{
        "kv_data_quant_config":{
            "act_algo":"hfmg",
            "num_of_bins":4096,
            "quant_granularity":0
        }
    },
    "matmul3":{
        "kv_data_quant_config":{
            "act_algo":"ifmr",
            "max_percentile":0.999999,
            "min_percentile":0.999999,
            "search_range":[
                0.7,
                1.3
            ],
            "search_step":0.01,
            "quant_granularity":0
        }
    }
}

Parent topic: KV Cache Quantization APIs