eval

Feature Overview

• Feature entry: $HOME/.local/lib/python3.x/site-packages/amct_pytorch/cli/llm/eval.py
• Description: This entry performs PPL evaluation and supports two evaluation modes.
  • BF16 benchmark performance evaluation: Loads blocks of the original model and computes the PPL in BF16 precision (--eval_mode=bf16, where eval_mode specifies the evaluation mode).
  • Quantization performance evaluation: Builds quantization blocks, enables the quantizer based on --bit_config (path to the configuration file in YAML format) and --quant_target (quantization target), and computes the PPL of the quantized model (--eval_mode=quant).
• Output: Generates evaluation logs containing PPL values in the output_dir/logs/ directory.
• Constraints: Considering user-environment limitations, set --granularity (quantization granularity) to block whenever possible.

Instantiation operation 1: BF16 benchmark PPL measurement

Compute the PPL of the BF16 full-precision model, which serves as the evaluation benchmark for post-quantization accuracy. The command is as follows:

1

python3 -m amct_pytorch.eval --model /path/to/model --model_name qwen3 --device npu:0 --granularity block --eval_mode bf16 --bit_config amct_pytorch/configs/bf16.yaml

Note that --bit_config and --model must be set to local paths. For details about the parameters, see Parameters.

Instantiation operation 2: PPL measurement for the quantized model

Quantization evaluation rebuilds the quantization blocks and determines whether to enable the quantizer based on --bit_config. The command is as follows:

1

python3 -m amct_pytorch.eval --model /path/to/model --model_name qwen3 --device npu:0 --granularity block --eval_mode quant --quant_target mlp --quant_dtype int --bit_config amct_pytorch/configs/example_w4a8.yaml

Note that --bit_config and --model must be set to local paths. For details about the parameters, see Parameters.

Instantiation operation 3: PPL test for the quantized model with PTQ training results

Quantization evaluation rebuilds the quantization blocks and determines whether to enable the quantizer based on --bit_config. The command is as follows:

1

python3 -m amct_pytorch.eval --model /path/to/model --model_name qwen3 --device npu:0 --granularity block --eval_mode quant --quant_target mlp --algos lwc lac --quant_dtype int --bit_config amct_pytorch/configs/example_w4a8.yaml --moe_mlp_param_dir ./outputs/qwen3_ptq_params/mlp

For details about the parameters, see Parameters.