部署命令如下,910b 8卡,推理速度为14tokens/s,不使用量化模型的话,如何提升推理速度
docker run -itd \
> --name qwen36-27b \
> --net=host \
> --shm-size=512g \
> --device /dev/davinci0 \
> --device /dev/davinci1 \
> --device /dev/davinci2 \
> --device /dev/davinci3 \
> --device /dev/davinci4 \
> --device /dev/davinci5 \
> --device /dev/davinci6 \
> --device /dev/davinci7 \
> --device /dev/davinci_manager \
> --device /dev/devmm_svm \
> --device /dev/hisi_hdc \
> -v /usr/local/dcmi:/usr/local/dcmi \
> -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
> -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
> -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
> -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
> -v /etc/ascend_install.info:/etc/ascend_install.info \
> -v /etc/hccn.conf:/etc/hccn.conf \
> -v /home/htm_test/Qwen3.6-27B:/data/models/qwen36-27b \
> -e ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
> quay.io/ascend/vllm-ascend:v0.18.0 \
> bash -lc "vllm serve /data/models/qwen36-27b \
> --host 0.0.0.0 \
> --port 8000 \
> --tensor-parallel-size 8 \
> --pipeline-parallel-size 1 \
> --max-num-seqs 1"
部署命令如下,910b 8卡,推理速度为14tokens/s,不使用量化模型的话,如何提升推理速度
docker run -itd \
> --name qwen36-27b \
> --net=host \
> --shm-size=512g \
> --device /dev/davinci0 \
> --device /dev/davinci1 \
> --device /dev/davinci2 \
> --device /dev/davinci3 \
> --device /dev/davinci4 \
> --device /dev/davinci5 \
> --device /dev/davinci6 \
> --device /dev/davinci7 \
> --device /dev/davinci_manager \
> --device /dev/devmm_svm \
> --device /dev/hisi_hdc \
> -v /usr/local/dcmi:/usr/local/dcmi \
> -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
> -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
> -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
> -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
> -v /etc/ascend_install.info:/etc/ascend_install.info \
> -v /etc/hccn.conf:/etc/hccn.conf \
> -v /home/htm_test/Qwen3.6-27B:/data/models/qwen36-27b \
> -e ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
> quay.io/ascend/vllm-ascend:v0.18.0 \
> bash -lc "vllm serve /data/models/qwen36-27b \
> --host 0.0.0.0 \
> --port 8000 \
> --tensor-parallel-size 8 \
> --pipeline-parallel-size 1 \
> --max-num-seqs 1"