使用kv cache w8a8量化qwen2-72b后模型显存占用和推理速度并没有很好的优化,请问是什么问题:
1、python examples/models/qwen/convert_quant_weights_72b.py \
--model_path /home/model/Qwen2-72B-Instruct \
--save_directory /work/Qwen2-72B-Instruct-w8a8kv \
--w_bit 8 \
--a_bit 8 \
--disable_level L0 \
--device_type npu \
--calib_file examples/convert/model_slim/boolq.jsonl \
--use_kvcache_quant True

2、 bash examples/models/qwen/run_pa.sh -m /work/Qwen2-72B-Instruct-w8a8kv
1、python examples/models/qwen/convert_quant_weights_72b.py \
--model_path /home/model/Qwen2-72B-Instruct \
--save_directory /work/Qwen2-72B-Instruct-w8a8kv \
--w_bit 8 \
--a_bit 8 \
--disable_level L0 \
--device_type npu \
--calib_file examples/convert/model_slim/boolq.jsonl \
--use_kvcache_quant True

2、 bash examples/models/qwen/run_pa.sh -m /work/Qwen2-72B-Instruct-w8a8kv