Simplified KV Cache Quantization Configuration File

The following table describes the parameters in the quant_calibration_config_pytorch.proto file. Find the file in /amct_pytorch/proto/ under the AMCT installation directory.

**Table 1** quant_calibration_config_pytorch.proto
Parameter	Required/Optional	Type	Field	Description
AMCTQuantCaliConfig	-	-	-	Simplified configuration of AMCT KV cache quantization.
	Optional	UInt32	batch_num	Number of batches used for quantization, which is used by the IFMR/HFMG quantization algorithm to accumulate data and calculate quantization factors.
	Optional	Boolean	activation_offset	Whether to quantize activations with offset. It is a global configuration parameter. true: with offset. Activations are asymmetrically quantized. false: without offset. Activations are symmetrically quantized.
	Optional	CommonCalibrationConfig	kv_cache_quant_config	Common quantization configuration for KV cache, which is a global parameter. This configuration is used if a layer is not overridden by override_layer_configs. Parameter priority: override_layer_configs > kv_cache_quant_config
	Repeated	OverrideLayer	override_layers_configs	Layer to override the quantization configurations. It is used to determine which layers are to be differentiatedly quantized. By using this parameter, you can perform differentiated quantization on some layers to change the quantization factor search step from 0.01 to 0.02. Parameter priority: override_layer_configs > kv_cache_quant_config
CommonCalibrationConfig	-	-	-	Common quantization configuration for KV cache.
	Repeated	String	quant_layers	Layers that support quantization.
	Optional	CalibrationConfig	calibration_config	Quantization configuration.
OverrideLayer	-	-	-	Quantization configuration to override by layer.
	Repeated	String	layer_name	Layers to override.
	Optional	CalibrationConfig	kv_data_quant_config	KV cache quantization configuration to override.
CalibrationConfig	-	-	-	Calibration-based quantization configuration.
	-	FMRQuantize	ifmr_quantize	Activation quantization algorithm configuration. ifmr_quantize: IFMR algorithm configuration.
	-	HFMGQuantize	hfmg_quantize	Activation quantization algorithm configuration. hfmg_quantize: HFMG algorithm configuration.
FMRQuantize	-	-	-	FMR algorithm for activation quantization. For details about the algorithm, see IFMR Algorithm. This parameter is mutually exclusive with HFMGQuantize.
	Optional	Float	search_range_start	Quantization factor search start.
	Optional	Float	search_range_end	Quantization factor search end.
	Optional	Float	search_step	Quantization factor search step.
	Optional	Float	max_percentile	Upper bound for searching for the largest.
	Optional	Float	min_percentile	Lower bound for searching for the smallest.
	Optional	Boolean	asymmetric	Whether to perform symmetric quantization. It is used to select the layer-wise quantization algorithm. true: asymmetric quantization false: symmetric quantization If this parameter is set for override_layer_configs and common_config, or if the activation_offset parameter is set, the priority is as follows: override_layer_configs>common_config>activation_offset
	Optional	QuantGranularity	quant_granularity	Quantization granularity. The options are as follows: 0 (default): per_tensor. 1: per_channel.
HFMGQuantize	-	-	-	HFMG algorithm for activation quantization. For details about the algorithm, see HFMG Algorithm. This parameter is mutually exclusive with FMRQuantize.
	Optional	UInt32	num_of_bins	Number of bins (the minimum unit in a histogram). Value range: {1024, 2048, 4096, 8192}. Defaults to 4096.
	Optional	Boolean	asymmetric	Whether to perform symmetric quantization. It is used to select the layer-wise quantization algorithm. true: asymmetric quantization false: symmetric quantization If this parameter is set for override_layer_configs and common_config, or if the activation_offset parameter is set, the priority is as follows: override_layer_configs>common_config>activation_offset
	Optional	QuantGranularity	quant_granularity	Quantization granularity. The options are as follows: 0 (default): per_tensor. 1: per_channel.

The following is an example of the simplified configuration file (quant.cfg) for KV cache quantization:

kv_cache_quant_config {
    quant_layers: 'matmul1'
    quant_layers: 'matmul2'
    calibration_config: {
        hfmg_quantize : {
        }
    }
}

override_layers_configs {
    layer_name: 'matmul3'
    kv_data_quant_config: {
        ifmr_quantize : {
        }
    }
}

Parent topic: Reference