Activation Quantization Balance Preprocessing

In scenarios where activations are unevenly distributed, the result of per-tensor quantization on activations has a large error due to outliers, while per-channel quantization result has a small error. The current hardware does not support per-channel quantization for activations and only supports per-channel quantization for weights. To reduce the quantization error, this section introduces a special method based on AMCT.

Use the activation quantization balance preprocessing API to calculate the balance factor, perform a mathematical equivalent conversion between the model activations and weights to balance their distribution, and then migrate some of the quantization difficulties from activations to weights. The layers supported by this feature as well as their constraints are listed as follows.

**Table 1** Supported layers and constraints
Supported Layer Type	Constraints	Remarks
torch.nn.Linear	-	Layers sharing the weight and bias parameters do not support quantization.
torch.nn.Conv2d	padding_mode = zeros
torch.nn.Conv3d	dilation_d = 1, dilation_h/dilation_w ≥ 1 padding_mode = zeros
torch.nn.ConvTranspose2d	padding_mode = zeros

API Call Sequence

Figure 1 shows the API call sequence for balance preprocessing.

Figure 1 API call sequence for balance preprocessing

The user implements the operations in blue, while those in gray are implemented by using AMCT APIs. The tool usage scenarios are as follows:

Build the original PyTorch model, set the DMQ parameters in the simplified configuration file dmp_quant.cfg (for details about the parameters in the simplified configuration file, see DMQBalancer in Simplified PTQ Configuration File), and pass the configuration file to the create_quant_config API.
Optimize the original PyTorch model using the quantize_preprocess API based on the quantization configuration file. The optimized model contains balanced quantization algorithms.
Calibrate the model by running forward passes once on the calibration dataset in the PyTorch environment to obtain the balanced quantization factor and save it into a file.
Optimize the original PyTorch model again using the quantize_model API based on the PyTorch model, quantization configuration file, and record file. The optimized model contains quantization algorithms.
Calibrate the model by running forward passes on the calibration dataset in the PyTorch environment to obtain the quantization factor and save it into a file.
Call the save_model API to save the quantized model, including the fake-quantized model file in the ONNX Runtime environment or deployable model file on the AI processor.

Example

Take the following steps to get started. Update the sample code based on your situation.
Tweak the arguments passed to AMCT API calls as required.

Import the AMCT package and set the log level using the environment variable in "AMCT (PyTorch)" in Post-installation Actions.
1
import amct_pytorch as amct
(Optional) Validate the inference script and environment setup in the source PyTorch environment. (Update the sample code based on your situation.)
You are advised to run inference on the original model in the PyTorch environment based on the test dataset to validate the environment setup and inference script.

This step is recommended as it guarantees a properly functioning original model for inference with acceptable accuracy. You can use a subset from the test dataset to improve the efficiency.
1
user_do_inference_torch(ori_model, test_data, test_iterations)

Run AMCT to quantize the model.

Generate a quantization configuration file.

You can set the DMQ parameters in the simplified configuration file dmp_quant.cfg and pass the configuration file to the create_quant_config API through the config_defination parameter.

config_defination = os.path.join(PATH, 'dmp_quant.cfg')
config_file = './tmp/config.json'
skip_layers = []
batch_num = 1
amct.create_quant_config(config_file=config_file,
			        model=ori_model,
                                input_data=ori_model_input_data,
			        skip_layers=skip_layers,
			        batch_num=batch_num,
                                config_defination=config_defination)

Modify the graph by inserting the balanced quantization operator to calculate the balanced quantization factors.

record_file = './tmp/record.txt'
modified_onnx_model = './tmp/modified_model.onnx'
calibration_model = amct.quantize_preprocess(config_file=config_file,
					            record_file=record_file,
                                                    model=ori_model,
                                                    input_data=ori_model_input_data)

Run inference on the modified model (calibration_model) in the PyTorch environment based on the calibration dataset (calibration_data) to determine the balanced quantization factor. (Update the sample code based on your situation.)
Pay attention to the following points:
1. Ensure that the calibration dataset and the preprocessed data match the model to preserve the accuracy.
2. Ensure that the number of forward passes is 1. If the number exceeds 1, the subsequent process will fail as the balanced quantization factor is recorded each time inference is executed.
1
user_do_inference_torch(calibration_model, calibration_data, test_iterations=1)

Modify the graph by inserting activation and weight quantization operators for quantization parameter calculation.

modified_onnx_model = './tmp/modified_model.onnx'
calibration_model = amct.quantize_model(config_file=config_file,
					       modified_onnx_model=modified_onnx_model,
					       record_file=record_file,
                                               model=ori_model,
                                               input_data=ori_model_input_data)

Run inference on the modified model (calibration_model) in the PyTorch environment based on the calibration dataset (calibration_data) to determine the quantization factors. (Update the sample code based on your situation.)
Pay attention to the following points:
1. Ensure that the calibration dataset and the preprocessed data match the model to preserve the accuracy.
2. Ensure that the number of forward passes (specified by batch_num) is large enough. Otherwise, the subsequent process will fail.
1
user_do_inference_torch(calibration_model, calibration_data, batch_num)

Save the model.

Call the save_model API to insert operators such as AscendQuant and AscendDequant and save the quantized models based on the quantization factors.

quant_model_path = './results/user_model'
amct.save_model(modified_onnx_file=modified_onnx_file,
                       record_file=record_file,
                       save_path=quant_model_path)

(Optional) Run inference on the fake-quantized model (quant_model) in the ONNX Runtime environment based on the test dataset (test_data) to test the accuracy. (Update the sample code based on your situation.)
Check the accuracy drop (from quantization) of the fake-quantized model by comparing with that of the original model in 2.
1 2
quant_model = './results/user_model_fake_quant_model.onnx' user_do_inference_onnx(quant_model, test_data, test_iterations)

Parent topic: Graph-based Quantization