deploy

Feature Overview

• Feature entry: $HOME/.local/lib/python3.x/site-packages/amct_pytorch/cli/llm/deploy.py
• Description: This entry exports weights for deployment. The workflow is as follows:
  1. Base safetensors file loading: The system loads the original safetensors weight index from the source file for subsequent operations.
  2. Quantization module build: The system builds block modules for quantization processing based on the specified PTQ parameters.
  3. Tensor export: The system generates and exports the quantized weight tensor and related tensor data such as scale parameters for deployment.
  4. Unreplaced weight management: The original weights that are not processed during quantization are reshared and stored in the rest_*.safetensors file for restoration or further use.
  5. Configuration file update: Upon completion, the system updates the core configuration files model.safetensors.index.json and config.json to ensure file integrity and consistency.
• Output: Generates a deployable model directory containing layer_*.safetensors, rest_*.safetensors, and updated model.safetensors.index.json and config.json.
• Constraints:
  • Module granularity: Only processing at block granularity (--granularity=block) is supported currently.
  • Custom hook limitation: If a custom quantize() hook (for weight quantization algorithms) is used, deployment via export_deploy() is not supported in the current version.

Instantiation Operation: Exporting Weights for Deployment

deploy supports export at block granularity, during which the non-weight auxiliary files of the model are copied, quantized weights are exported layer by layer, model.safetensors.index.json is rewritten, and quantization_config is written into config.json. The command is as follows:

1

python3 -m amct_pytorch.deploy --model /path/to/model --model_name qwen3 --granularity block --quant_target mlp --quant_dtype mxfp --bit_config amct_pytorch/configs/example_w4a8.yaml --moe_mlp_param_dir ./outputs/qwen3_ptq_params/mlp --output_dir ./outputs/qwen3_deploy