Activation Quantization Balance Preprocessing

In scenarios where activations are unevenly distributed, the result of per-tensor quantization on activations has a large error due to outliers, while per-channel quantization result has a small error. The current hardware does not support per-channel quantization for activations and only supports per-channel quantization for weights. To reduce the quantization error, this section introduces a special method based on AMCT.

Use the activation quantization balance preprocessing API to calculate the balance factor, perform a mathematical equivalent conversion between the model activations and weights to balance their distribution, and then migrate some of the quantization difficulties from activations to weights.

**Table 1** Supported layers and restrictions
Supported Layer Type	Restriction	Remarks
Conv	Only quantization with 4D or 5D inputs is supported.	Layers sharing the weight do not support activation quantization balance preprocessing.
Gemm	transpose_a = false, Alpha = Beta = 1.0.
MatMul	Quantization is supported only when the weight has rank 2.
ConvTranspose	Only quantization with 4D inputs is supported.

API Call Sequence

Figure 1 shows the API call sequence for balance preprocessing.

Figure 1 API call sequence for balance preprocessing

The user implements the operations in blue, while those in gray are implemented by using AMCT APIs.

Prepare the original ONNX model, set the DMQ parameters in the simplified configuration file dmp_quant.cfg (for details about the parameters in the simplified configuration file, see DMQBalancer in Simplified PTQ Configuration File), and pass the configuration file to create_quant_config.
Call the quantize_preprocess API to optimize the original ONNX model before quantization based on the ONNX model and quantization configuration file. Balanced quantization migrates the quantization difficulties from activations to weights.
Calibrate the model by running a single forward pass on the calibration dataset in the ONNX Runtime environment to complete balanced quantization and save the balance factor to a record file.
Optimize the original ONNX model using the quantize_model API based on the quantization configuration file and record, including Conv+BN fusion. Next, insert weight and activation quantization operators for weight quantization and calibration.
Calibrate the model by running forward passes on the calibration dataset in the ONNX Runtime environment to complete activation quantization and save quantization factors to a file.
Call the save_model API to save the quantized model, including the fake-quantized model file in the ONNX Runtime environment or deployable model file on the AI processor.

Example

Take the following steps to get started. Update the sample code based on your situation.
Tweak the arguments passed to AMCT API calls as required.

Import the AMCT package and set the log level (see Setting Environment Variables for details).
1
import amct_onnx as amct
(Optional) Validate the inference script and environment setup in the source ONNX Runtime environment. (Update the sample code based on your situation.)
You are advised to run inference on the original model for quantization in the ONNX Runtime environment based on the test dataset to validate the environment setup and inference script.

This step is recommended as it guarantees a properly functioning original model for inference with acceptable accuracy. You can use a subset from the test dataset to improve the efficiency.
1
user_do_inference(ori_model, test_data, test_iterations)

Run AMCT to quantize the model.

Generate a quantization configuration file.

You can set the DMQ parameters in the simplified configuration file dmp_quant.cfg and pass the configuration file to the create_quant_config API through the config_defination parameter.

config_defination = os.path.join(PATH, 'dmp_quant.cfg')
config_file = './tmp/config.json'
skip_layers = []
batch_num = 1
amct.create_quant_config(config_file=config_file,
			        model=ori_model,
                                input_data=ori_model_input_data,
			        skip_layers=skip_layers,
			        batch_num=batch_num,
                                config_defination=config_defination)

Modify the graph by inserting the balanced quantization operator to calculate the balanced quantization factors.

record_file = './tmp/record.txt'
modified_model = './tmp/modified_model.onnx'
amct.quantize_preprocess(config_file=config_file,
                               record_file=record_file,
                               model_file=ori_model,
                               modified_onnx_file=modified_model)

Run inference on the modified graph based on the calibration dataset to calculate the balanced quantization factors. (Update the sample code based on your situation.)
Pay attention to the following points:
1. Ensure that the calibration dataset and the preprocessed data match the model to preserve the accuracy.
2. Ensure that the number of forward passes is 1. If the number exceeds 1, the subsequent process will fail as the balanced quantization factor is recorded each time inference is executed.
1
user_do_inference(modified_onnx_file, calibration_data, test_iterations=1)

Modify the graph by inserting activation and weight quantization operators for quantization parameter calculation.

modified_model = './tmp/modified_model.onnx'
amct.quantize_model(config_file=config_file,
                          model_file=ori_model,
                          modified_onnx_file=modified_model,
                          record_file=record_file)

Run inference on the modified graph based on the calibration dataset to determine the quantization factors. (Update the sample code based on your situation.)
Pay attention to the following points:
1. Ensure that the calibration dataset and the preprocessed data match the model to preserve the accuracy.
2. Ensure that the number of forward passes (specified by batch_num) is large enough. Otherwise, the subsequent process will fail.
Refer to What Do I Do If "IFMR node. Name:'layer_ifmr_op' Status Message: std::bad_alloc" Is Displayed During the Calibration? or Why Am I Seeing the "killed" Message During Calibration? for troubleshooting.
1
user_do_inference(modified_onnx_file, calibration_data, batch_num)

Save the model.

Call the save_model API to insert operators such as AscendQuant and AscendDequant and save the quantized models based on the quantization factors.

quant_model_path = './results/user_model'
amct.save_model(modified_onnx_file=modified_model,
                      record_file=record_file,
                      save_path=quant_model_path)

(Optional) Run inference on the fake-quantized model (quant_model) in the ONNX Runtime environment based on the test dataset (test_data) to test the accuracy. (Update the sample code based on your situation.)
Check the accuracy drop (from quantization) of the fake-quantized model by comparing with that of the original model in 2.
1 2
quant_model = './results/user_model_fake_quant_model.onnx' user_do_inference(quant_model, test_data, test_iterations)

Parent topic: More Features