Overview

This section describes the usage scenarios of torch module-based PTQ.

This feature is supported only by the following products. Ensure that PyTorch 2.1.0 is used. For details, see Table 1.

Atlas 350 Accelerator Card

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product

Scenario

The differences between torch module-based PTQ and Graph-based Quantization are as follows:

Graph-based compression requires the PyTorch model to be exportable to ONNX. It enables graph fusion operations such as Conv+BN and Matmul+Add, resulting in better network performance after compression. In contrast, torch module-based PTQ is not subject to these constraints.

Prerequisites

All operators in the model to be quantized must be supported by the NPU. If unsupported operators are present, the quantized deployable model will fail to infer on the NPU. To verify operator support, ensure that the PyTorch training script can run successfully on the NPU before quantization. For details, see PyTorch Model Porting and Tuning Guide.

Parent topic: Torch Module-based Quantization