Overview
Compression combination is used to both quantize and sparsify a network model. This section describes the supported compression combinations.
Compression Combination
The following compression combination modes are currently supported. When AMCT is used for compression, only one compression combination mode can be selected for each compressible operator at a time. Figure 1 shows the process.
- Manual Sparsity + Quantization Aware Training for INT8 quantization
- 2:4 Structured Sparsity + Quantization Aware Training for INT8 quantization
Currently, the compression combination feature requires manual compression configuration (also referred to as static compression combination). Set global quantization data type and sparsity ratio (the ratio of the number of pruned filters to the total number of filters) or interval for updating 2:4 sparsity for automatic model compression. For details about the compression configuration file, see Simplified QAT Configuration File. For examples of compression combination, see Sample List.
For the layers that support compression combination as well as their restrictions, see Table 1 in Manual Sparsity, Table 1 in 2:4 Structured Sparsity, or Table 1 in Quantization Aware Training.
Introduction to Compression Combination Scenarios
Currently, compression combination is supported in the following scenarios. In practice, you can configure parameters in the simplified configuration file to control the compression combination. In the following scenarios, the network-wide quantization refers to QAT specified in Quantization Aware Training, where:
- Network-wide quantization:
- Network-wide (global) quantization configuration parameter: retrain_data_quant_config or retrain_weight_quant_config
- Differentiated configuration parameter for some layers: override_layer_configs or override_layer_types
Parameter priority: override_layer_configs > override_layer_types > retrain_data_quant_config/retrain_weight_quant_config
- Network-wide sparsity: includes filter-level sparsity and 2:4 structured sparsity. Use either of them.
- Network-wide (global) sparsity configuration parameter: prune_config
- Differentiated sparsity parameter for some layers: override_layer_configs or override_layer_types
Parameter priority: override_layer_configs > override_layer_types > retrain_data_quant_config/retrain_weight_quant_config
For details about the preceding parameters, see Simplified QAT Configuration File.
Scenario |
Parameter |
Description |
|---|---|---|
Network-wide quantization + Network-wide sparsity |
Do not set override_layer_configs or override_layer_types. |
- |
Differentiated quantization for some layers + Network-wide sparsity |
|
Before setting feature parameters, specify the global configuration. Otherwise, the feature is disabled. |
Network-wide quantization + Differentiated sparsity of some layers |
|
Before setting feature parameters, specify the global configuration. Otherwise, the feature is disabled. |
Quantization of some layers + Differentiated sparsity of some layers |
|
Before setting feature parameters, specify the global configuration. Otherwise, the feature is disabled. |
Network-wide quantization |
retrain_data_quant_config/retrain_weight_quant_config Do not set override_layer_configs or override_layer_types. |
- |
Differentiated quantization of some layers |
retrain_data_quant_config/retrain_weight_quant_config + override_layer_configs or override_layer_types |
Before setting feature parameters, specify the global configuration. Otherwise, the feature is disabled. |
Network-wide sparsity |
prune_config Do not set override_layer_configs or override_layer_types. |
- |
Differentiated sparsity of some layers |
prune_config + override_layer_configs or override_layer_types |
Before setting feature parameters, specify the global configuration. Otherwise, the feature is disabled. |