910B部署Ornith-1.0-35B为什么推理很慢
收藏回复举报
910B部署Ornith-1.0-35B为什么推理很慢
t('forum.solved') 已解决
发表于2026-07-01 12:07:31
0 查看

2张910B 64G,推理速度如下,20k上下文为什么不到25t/s:

cke_9128.png

部署命令如下:

export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0

export NAME=o35

docker run --rm \

--name $NAME \

--net=host \

--shm-size=100g \

--ulimit nproc=65535:65535 \

--pids-limit=8192 \

--security-opt seccomp=unconfined \

--device /dev/davinci2 \

--device /dev/davinci3 \

--device /dev/davinci_manager \

--device /dev/devmm_svm \

--device /dev/hisi_hdc \

-v /usr/local/dcmi:/usr/local/dcmi \

-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \

-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \

-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \

-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \

-v /etc/ascend_install.info:/etc/ascend_install.info \

-v /home/htm_test:/root/.cache \

-it $IMAGE bash

export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

export HCCL_OP_EXPANSION_MODE="AIV"

export HCCL_BUFFSIZE=1024

export OMP_NUM_THREADS=1

export TASK_QUEUE_ENABLE=1

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

export VLLM_TARGET_DEVICE=ascend

vllm serve /root/.cache/Ornith-1.0-35B \

    --served-model-name qwen3.6-27b\

    --tensor-parallel-size 2 \

    --host 0.0.0.0 --port 8011 \

    --max-model-len 262144 \

    --gpu-memory-utilization 0.90 \

    --enable-auto-tool-choice --tool-call-parser qwen3_xml \

    --reasoning-parser qwen3 \

    --trust-remote-code

我要发帖子