Overview

This section describes the quantization tool for large language models (LLMs) in amct_pytorch. Centered on a command-line workflow, the tool orchestrates key capabilities including perplexity (PPL) measurement, PTQ data extraction, PTQ parameter training, and export of quantized weights for deployment. Meanwhile, with the model adapter, quantization algorithm registry, and quantization data type registry, flexible combinations of different model architectures and quantization policies are realized.

Tool Introduction

The preceding command-line tool is installed in the $HOME/.local/lib/python3.x/site-packages/amct_pytorch directory (using a non-root user as an example). The tool consists of the following modules. For details about the end-to-end example, see Qwen-3.6-MoE One-Stop Example.

Module	Path	Description
CLI entry	amct_pytorch/cli/llm/	Provides four types of CLI entries: eval: Evaluates the PPL. extract_ptq_data: Extracts PTQ calibration data. ptq: Performs PTQ optimization training. deploy: Exports weights for deployment.
Workflow	amct_pytorch/workflows/	Supports orchestration of PPL evaluation, calibration data extraction, PTQ optimization training, and weight export for deployment.
Model adapter	amct_pytorch/common/models/llm/	Adapts to model structures such as DeepSeek, Qwen, LongCat, Pangu, and GLM.
Quantization algorithm	amct_pytorch/algorithms/quant/	Provides algorithm components such as LWC, LAC, OmniQuant, Learnable Hadamard, and AutoRound.
Quantization data type	amct_pytorch/quantization/dtypes/	Implements quantization and dequantization for data types such as mxfp and int.

Typical supported capabilities:

Benchmark evaluation: Computes the PPL of BF16 or quantized models on the WikiText dataset.
Data extraction: Efficiently extracts input activations and intermediate-layer outputs required for PTQ calibration using the Pileval dataset.
Block-level optimization: Performs PTQ parameter training and optimization for a specified quantization target at the block granularity.
Model export: Outputs the safetensors weight file and corresponding quantization configuration information that adapt to the deployment environment.
Flexible configuration: Sets the number of quantization bits for global or group-specific weights/activations (W/A) using bit_config.
Algorithm selection: Freely combines and selects trainable quantization algorithm policies using the algos parameter.

Environment Setup

Ensure that the model directory uses the HuggingFace/safetensors format and contains config.json and tokenizer files, as well as model.safetensors.index.json. Supported model adapters and their corresponding LLMs are listed as follows.

Model Name (--model_name)	Description
deepseek_v3_2	DeepSeek V3.2
deepseek_v4	DeepSeek V4
qwen3	Qwen3 Dense
qwen3_moe	Qwen3 MoE
qwen3_5	Qwen3.5 Dense
qwen3_5_moe	Qwen3.5 MoE
qwen3_6_moe	Qwen3.6 MoE
qwen3_next	Qwen3 Next
longcat_lite	LongCat Flash Lite
longcat_next	LongCat Next
glm5	GLM-5.1

When downloading the preceding LLMs, ensure that the server has network connectivity, then run the following commands in sequence. The following procedure is for reference only; you may adopt other download approaches.

Install dependencies (uses a non-root user as an example):
```
pip3 install  huggingface_hub --user
```
Download the model.
- Click the link in the model to open its model page, click the copy button to obtain the model name, as shown in the following figure.
- Run the following command to download the model:
```
huggingface-cli download --resume-download deepseek-ai/DeepSeek-V4-Pro --local-dir ./your_path
```
  Where:
  - --resume-download: specifies the name of the model to be downloaded.
  - --local-dir: specifies the local path for storing the model. Relative paths are supported.

Parent topic: LLM-based Quantization