AMCT (PyTorch)

OS and version

Ubuntu 20.04 (x86_64)

Download the required software release from the Ubuntu official website and install it. Run the following command to query the installation:

cat /etc/*release && uname -m

• Quantization on the CPU or NPU is supported.
• If the PyTorch-based AMCT package is installed, eight threads are used for building by default. Therefore, the OS running memory must be greater than or equal to 12 GB.

OS and version

Ubuntu 20.04 (AArch64)

Download the required software release from the Ubuntu official website and install it. Run the following command to query the installation:

cat /etc/*release && uname -m

• Quantization on the CPU or NPU is supported.
• If the PyTorch-based AMCT package is installed, eight threads are used for building by default. Therefore, the OS running memory must be greater than or equal to 12 GB.

PyTorch

2.7.1, 2.1.0, 1.10.0, 1.8.0, 1.5.0, 1.4.0

• PyTorch 2.7.1 works with ONNX 1.18.0, ONNX Runtime 1.20.0, and Python 3.12/3.13.
• PyTorch 2.7.1 works with ONNX 1.16.1, ONNX Runtime 1.20.0, and Python 3.11.x.
• PyTorch 2.1.0 works with ONNX 1.14.0, ONNX Runtime 1.16.0, and Python 3.10.0 or Python3.11.x.
• PyTorch 1.10.0 works with ONNX 1.9.0, ONNX Runtime 1.8.0, and Python 3.9.2.
• PyTorch 1.8.0, 1.5.0, and 1.4.0 work with ONNX 1.8.0, ONNX Runtime 1.6.0, and Python 3.7.5.

Install the CPU or GPU version as needed. For details, see Installing Dependencies.

• Version 1.4.0 is not recommended due to disturbing warnings.
• PyTorch 1.8.0 and earlier versions do not support compression of models whose size is greater than or equal to 2 GB.
• PyTorch 2.1.0 and earlier versions do not support the feature specified in Torch Module-based Quantization.

NPU environment

Prepare a hardware environment with the AI processor and install the driver, firmware, and CANN package. For details, see CANN Software Installation.

You need to install the driver, firmware, and CANN package to perform quantization on the NPU.

Ascend Extension for PyTorch

For details about the version mapping, click here and see "Release Notes > Product Version Mapping."

-

• To perform LLM-based Quantization and Torch Module-based Quantization, you must install this dependency after the NPU environment is set up.
• When using this plugin, ensure that it matches the PyTorch version.

ONNX

1.18.0, 1.16.1, 1.14.0, 1.9.0, 1.8.0

For details, see Installing Dependencies.

• Ensure that the server has Internet access when installing dependencies.
• This document uses Python 3.9.2 as an example. The environment variables and installation commands are subject to the actual Python version.
• For known issues of ONNX Runtime 1.8.0, click here.
• Python 3.7.x and 3.8.x have reached EOL, and CANN will no longer support them. Upgrade your Python version to 3.9.x or later.

ONNX Runtime

1.20.0, 1.16.0, 1.8.0, 1.6.0

Python

Python3.12/3.13, Python3.11.x, Python3.10.x, Python3.9.x, Python3.8.x, Python3.7.x

For the Ubuntu OS, see Python 3.9.2 Installation on Ubuntu.

For EulerOS, see Python3.9.2 Installation on EulerOS.

NumPy

• PyTorch 2.7.1 and Python 3.12/3.13 work with NumPy 2.0.0 and later versions.
• PyTorch 2.7.1 and Python 3.11.x work with NumPy 1.24.2–1.26.4.
• PyTorch 2.1.0 works with NumPy 1.21.6 and later versions.
• PyTorch 1.10.0, 1.8.0, 1.5.0, and 1.4.0 work with NumPy 1.20.0–1.23.5.

For details, see Installing Dependencies.

Protobuf

• PyTorch 2.7.1 and Python 3.12/3.13 work with Protobuf 3.20.3.
• PyTorch 2.7.1 and Python 3.11.x work with Protobuf 3.20.2–3.20.3.
• PyTorch 2.1.0, 1.10.0, 1.8.0, 1.5.0, and 1.4.0 work with Protobuf 3.20.2 and later versions.

pyyaml

-

-

Install this dependency only when performing LLM-based quantization.

transformers

5.5.4

-

Install this dependency only when performing LLM-based quantization.

loguru

-

-

Install this dependency only when performing LLM-based quantization.

datasets

-

-

Install this dependency only when performing LLM-based quantization.

accelerate

-

-

Install this dependency only when performing LLM-based quantization.

compressed_tensors

0.15.0.1

-

Install this dependency only when performing LLM-based quantization.

scipy

-

-

Install this dependency only when performing LLM-based quantization.

torchao

-

-

Install this dependency only when performing LLM-based quantization.

einops

-

-

Install this dependency only when performing LLM-based quantization.

sentencepiece

-

-

Install this dependency only when performing LLM-based quantization.

datasets

4.8.4

-

Install this dependency only when performing LLM-based quantization.

zstandard

0.25.0

-

Install this dependency only when performing LLM-based quantization.