create_quant_cali_model
Applicable Products
Product |
Supported |
|---|---|
√ |
|
√ |
|
√ |
|
x |
|
√ |
|
x |
Note: For the products marked with x, no error is reported when the API is called, but no performance gains are obtained.
Description
Modifies the graph of the user's network model based on the model and detailed quantization configuration, replaces the Linear operator to be quantized with the output IFMR/HFMG quantization operator, calibrates the model online, and generates the quantization factors to be saved in record_file.
Prototype
1 | calibration_model = create_quant_cali_model(config_file, record_file, model) |
Parameters
Parameter |
Input/Output |
Description |
|---|---|---|
config_file |
Input |
Path of the generated quantization configuration file (JSON). A string. Restrictions: The config.json file passed to this API must be the same as that passed to the create_quant_cali_config API. |
record_file |
Input |
Path (including the file name) of the quantization factor record file saved during online quantization factor calibration. A string. |
model |
Input |
Original model. A torch.nn.Module. |
Returns
Replaced with the quantization calibration model of the calibration operator.
Example
1 2 3 4 5 6 7 8 9 10 11 | import amct_pytorch as amct # Build a graph of the network for quantization. model = build_model() model.load_state_dict(torch.load(state_dict_path)) record_file = os.path.join(TMP, 'kv_cache.txt') # Insert the quantization API to generate a quantization calibration model. calibration_model = amct.create_quant_cali_model( config_file="./configs/config.json", # Quantization factor record file. record_file, model) |