--compression_optimize_conf

Applicable Products

• Ascend 950PR/Ascend 950DT: not supported
• Atlas A3 training products/Atlas A3 inference products: supported
• Atlas A2 training products/Atlas A2 inference products: supported
• Atlas 200I/500 A2 inference products: supported
• Atlas inference products: supported
• Atlas training products: supported

Description

Sets the directory (including the file name) of the compression optimization configuration file. This option is used to enable the compression optimization feature specified in the configuration file to improve network performance.

See Also

If this option is used to configure the calibration quantization feature, the high-precision feature cannot be used. For example, force_fp32 or must_keep_origin_dtype (source image fp32 input) cannot be configured by --precision_mode, origin cannot be configured by --precision_mode_v2, and high_precision cannot be configured by --op_precision_mode. When quantization parameters are set in high-precision mode, neither the performance benefits of quantization nor the precision benefits of the high-precision mode can be obtained.

Arguments

Argument: Directory of the configuration file, including the file name.

Format: The directory (including the file name) can contain letters, digits, underscores (_), hyphens (-), periods (.), and Chinese characters.

Restrictions:

Currently, only the following compression modes are supported, you can choose whichever meets your requirements.

enable_first_layer_quantization: true
calibration: 
{
    input_data_dir: ./data.bin,d2.bin
    input_shape: in:16,16;in1:16,16
    config_file: simple_config.cfg
    infer_soc: xxxxxx
    infer_aicore_num: 10
    infer_device_id: 0 
    infer_ip: x.x.x.x
    infer_port: 8000
    log: info
}

Note that:

• enable_first_layer_quantization: Determines whether to optimize the first-layer convolution of AIPP (AIPP is fused with the Quant operator prior to the first-layer convolution Conv2D of the quantized model). In the configuration file, the field before the colon (:) indicates the name of the compression optimization feature, and the field after the colon (:) indicates whether to enable this feature. true indicates enabled, and false (default) indicates disabled.

  When the enable_first_layer_quantization feature is enabled, performance gain can be obtained only when the AIPP+Conv2D structure exists in the network structure and --enable_small_channel in the atc command is set to 1. Quantized models may suffer accuracy loss. You can decide whether to enable this feature as required.

  Only Atlas inference products and Atlas 200I/500 A2 inference products support this feature.

• calibration: Post-training quantization (PTQ) refers to quantizing the weights of an already trained model from floating-point numbers (float32 and float16 are supported currently) to low-bit integers (such as int8) as well as quantizing the activations by using a small calibration dataset, accelerating model inference. PTQ is easy to use and requires only a small calibration dataset, which is applicable to scenarios where ease of use and resource saving take priority. To obtain the PTQ sample, click here.
  Option description:

  • input_data_dir: (mandatory) directory containing the .bin file of model input calibration data. If the model has multiple inputs, use commas (,) to separate the input .bin data files. The calibration dataset is used to calculate quantization parameters. The calibration dataset must be representative. Subsets of the test dataset are recommended as the calibration dataset. For details about how to generate the binary file of the calibration data, see Link.
  • input_shape: (mandatory) shape information of the model input calibration data, for example, input_name1:n1,c1,h1,w1;input_name2:n2,c2,h2,w2. Use semicolons (;) to separate the nodes.
  • config_file: (optional) simplified PTQ configuration file. For details about the configuration example and option description of this file, see Simplified Quantization Configuration File.
  • infer_soc: (mandatory) name of the processor used for PTQ calibration and inference. For details about how to query the processor name, see Arguments.
  • infer_aicore_num: (optional) number of AI Cores used for PTQ calibration and inference. For details about the query method, see --aicore_num.
  • infer_device_id: (optional) ID of the device powered by the AI processor, which is used for PTQ calibration and inference. The default value is 0.
  • infer_ip: IP address of the server where the NCS software package is located. This parameter is mandatory in the Atlas 200I/500 A2 inference products Ascend RC scenario.
  • infer_port: port of the server where the NCS package is located. This parameter is mandatory in the Atlas 200I/500 A2 inference productsAscend RC scenario.
  • log: (optional) level of logs printed during PTQ. The default log level is info.
    • debug: Outputs debug, info, warning, error, and event logs.
    • info: Outputs info, warning, error, and event logs.
    • warning: Outputs warning, error, and event logs.
    • error: Outputs error and event logs.
    In addition, the AMCT_LOG_DUMP environment variable controls the log printing and flushing information during PTQ.

    • export AMCT_LOG_DUMP=1: Prints logs to the screen and flushes logs to the amct_log_timestamp/amct_acl.log file in the current path and does not save the quantization factor record file and graph file.
    • export AMCT_LOG_DUMP=2: Flushes logs to the amct_log_timestamp/amct_acl.log file in the current path and saves the quantization factor record file.
    • export AMCT_LOG_DUMP=3: Flushes logs to the amct_log_timestamp/amct_acl.log file in the current path and saves the quantization factor record file and graph file.

    To prevent the disk from being full due to continuous flushing of log files, record files, and graph files, delete these files in a timely manner.

    If the ASCEND_WORK_PATH environment variable is configured, the preceding logs, quantization factor record files, and graph files are stored in the path specified by this environment variable. For example, if ASCEND_WORK_PATH is set to /home/test, the storage path is /home/test/amct_acl/amct_log_{pid}_timestamp. amct_acl is automatically created during model conversion, and {pid} indicates the process ID.

    

    The preceding log files, record files, and graph files will be overwritten when quantization is performed again. You need to save them as required. In addition, the size of the generated log file is related to the number of layers of the model to be quantized. Ensure that the server where ATC is installed has sufficient space.

    Take the ResNet-101 model as an example. If the log level is set to INFO, the log file size is about 12 KB, and the size of the temporary file is about 260 MB. If the log level is set to DEBUG, the log file size is about 390 KB, and the size of the temporary file is about 430 MB.

Default: None

Suggestions and Benefits

None

Example

Assume that the configuration file of the compression function is compression_optimize.cfg. The configuration example is as follows:

enable_first_layer_quantization: true
calibration: 
{
    input_data_dir: ./data.bin,d2.bin
    input_shape: in:16,16;in1:16,16
    config_file: simple_config.cfg
    infer_soc: xxxxxx
    infer_aicore_num: 10
    infer_device_id: 0 
    infer_ip: x.x.x.x
    infer_port: 8000
    log: info
}

Upload the file to any directory (for example, $HOME/module) on the server where ATC is located.

--compression_optimize_conf=$HOME/module/compression_optimize.cfg

Dependencies and Restrictions

• To use the compression feature of this option, you need to install the AMCT (acl) software package separately. For details about how to obtain and install the package, see Environment Setup > Obtaining Software Packages/Uploading Software Packages/Installing Tools in AMCT.
• When the enable_first_layer_quantization feature is used, ensure that the model in use is the deployable model generated after quantization performed by AMCT.
• When the calibration feature in the configuration file is used, only installation scenarios with NPU-equipped devices are supported. For details, refer to CANN Software Installation to set up the product environment.
• In the Atlas 200I/500 A2 inference productsAscend RC scenario, you also need to install the NCS software in the operating environment and configure the key certificate. For details, see AOE (Ascend RC) > Environment Setup in AOE Tuning Tool.
• When the PTQ calibration feature in the configuration file is used, the ATC tool calls the AMCT quantization APIs to perform related operations. The following figure shows the schematic diagram.
  Figure 1 PTQ principles