我的设备是310P3，我想在设备上使用speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404。
我找到一个帖子如何在昇腾设备上部署paraformer模型：https://bbs.huaweicloud.com/blogs/438888
我按照上面帖子的格式部署，确实可以部署上也可以推理。
这是我的启动docker的命令：
sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
然后我在docker里面调用python推理脚本，但是我发现推理很慢，而且推理脚本好像并没有运行在npu上面。
我检查了docker的id：
ps aux | grep speech | grep -v grep
root 4000011 0.0 0.0 12972 4712 pts/0 S+ 08:13 0:00 sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
root 4000012 0.0 0.0 12972 700 pts/1 Ss 08:13 0:00 sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
root 4000013 0.0 0.0 1260596 22892 pts/1 Sl+ 08:13 0:00 docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
这是npu-info的信息：
npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.0.1 Version: 24.1.0.1 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310P3 | OK | NA 54 18116 / 18116 |
| 0 0 | 0000:01:00.0 | 0 37450/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 0 310P3 | OK | NA 53 17858 / 17858 |
| 1 1 | 0000:01:00.0 | 0 37491/ 43693 |
+===============================+=================+======================================================+
| 96 310P3 | OK | NA 58 18116 / 18116 |
| 0 2 | 0000:03:00.0 | 0 37744/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 96 310P3 | OK | NA 54 17858 / 17858 |
| 1 3 | 0000:03:00.0 | 0 37195/ 43693 |
+===============================+=================+======================================================+
| 32768 310P3 | OK | NA 52 19372 / 19372 |
| 0 4 | 0000:81:00.0 | 0 40221/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 32768 310P3 | OK | NA 53 13341 / 13341 |
| 1 5 | 0000:81:00.0 | 0 28290/ 43693 |
+===============================+=================+======================================================+
| 32896 310P3 | OK | NA 53 13341 / 13341 |
| 0 6 | 0000:84:00.0 | 0 28255/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 32896 310P3 | OK | NA 55 13341 / 13341 |
| 1 7 | 0000:84:00.0 | 0 28101/ 43693 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
| 0 0 | 26085 | mindie_llm_back | 36314 |
| 0 1 | 26087 | mindie_llm_back | 35798 |
+===============================+=================+======================================================+
| 96 0 | 26089 | mindie_llm_back | 36317 |
| 96 1 | 26096 | mindie_llm_back | 35801 |
+===============================+=================+======================================================+
| 32768 0 | 3930099 | mindie_llm_back | 26766 |
| 32768 0 | 19911 | python-text-emb | 822 |
| 32768 0 | 19178 | python-text-emb | 9011 |
| 32768 0 | 20142 | python-text-emb | 2474 |
| 32768 1 | 3930101 | mindie_llm_back | 26766 |
+===============================+=================+======================================================+
| 32896 0 | 3930103 | mindie_llm_back | 26765 |
| 32896 1 | 3930111 | mindie_llm_back | 26766 |
+===============================+=================+======================================================+
可以看到推理脚本是没有跑在npu上的。
我尝试了手动指定npu，比如export ASCEND_RT_VISIBLE_DEVICES=0 或者 在代码里显式指定 device="npu:0" ，好像都不能让推理脚本运行在npu上。
我想知道这种场景下如何让推理脚本或者说docker运行在npu上。
点赞
收藏
回复
分享
举报
已解决
我的设备是310P3，我想在设备上使用speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404。
我找到一个帖子如何在昇腾设备上部署paraformer模型：https://bbs.huaweicloud.com/blogs/438888
我按照上面帖子的格式部署，确实可以部署上也可以推理。
这是我的启动docker的命令：
sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
然后我在docker里面调用python推理脚本，但是我发现推理很慢，而且推理脚本好像并没有运行在npu上面。
我检查了docker的id：
ps aux | grep speech | grep -v grep
root 4000011 0.0 0.0 12972 4712 pts/0 S+ 08:13 0:00 sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
root 4000012 0.0 0.0 12972 700 pts/1 Ss 08:13 0:00 sudo docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
root 4000013 0.0 0.0 1260596 22892 pts/1 Sl+ 08:13 0:00 docker run -it -u=0 --net=host --privileged=true --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc --shm-size=512g -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /var/log/npu/:/usr/slog -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /data/.model/emb/speech1:/data/.model/emb/speech --name speech 243888a362d3 /bin/bash
这是npu-info的信息：
npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.0.1 Version: 24.1.0.1 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310P3 | OK | NA 54 18116 / 18116 |
| 0 0 | 0000:01:00.0 | 0 37450/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 0 310P3 | OK | NA 53 17858 / 17858 |
| 1 1 | 0000:01:00.0 | 0 37491/ 43693 |
+===============================+=================+======================================================+
| 96 310P3 | OK | NA 58 18116 / 18116 |
| 0 2 | 0000:03:00.0 | 0 37744/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 96 310P3 | OK | NA 54 17858 / 17858 |
| 1 3 | 0000:03:00.0 | 0 37195/ 43693 |
+===============================+=================+======================================================+
| 32768 310P3 | OK | NA 52 19372 / 19372 |
| 0 4 | 0000:81:00.0 | 0 40221/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 32768 310P3 | OK | NA 53 13341 / 13341 |
| 1 5 | 0000:81:00.0 | 0 28290/ 43693 |
+===============================+=================+======================================================+
| 32896 310P3 | OK | NA 53 13341 / 13341 |
| 0 6 | 0000:84:00.0 | 0 28255/ 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 32896 310P3 | OK | NA 55 13341 / 13341 |
| 1 7 | 0000:84:00.0 | 0 28101/ 43693 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
| 0 0 | 26085 | mindie_llm_back | 36314 |
| 0 1 | 26087 | mindie_llm_back | 35798 |
+===============================+=================+======================================================+
| 96 0 | 26089 | mindie_llm_back | 36317 |
| 96 1 | 26096 | mindie_llm_back | 35801 |
+===============================+=================+======================================================+
| 32768 0 | 3930099 | mindie_llm_back | 26766 |
| 32768 0 | 19911 | python-text-emb | 822 |
| 32768 0 | 19178 | python-text-emb | 9011 |
| 32768 0 | 20142 | python-text-emb | 2474 |
| 32768 1 | 3930101 | mindie_llm_back | 26766 |
+===============================+=================+======================================================+
| 32896 0 | 3930103 | mindie_llm_back | 26765 |
| 32896 1 | 3930111 | mindie_llm_back | 26766 |
+===============================+=================+======================================================+
可以看到推理脚本是没有跑在npu上的。
我尝试了手动指定npu，比如export ASCEND_RT_VISIBLE_DEVICES=0 或者 在代码里显式指定 device="npu:0" ，好像都不能让推理脚本运行在npu上。
我想知道这种场景下如何让推理脚本或者说docker运行在npu上。