我把4张910B3做了vnpu，虚拟化实例模板为vir10_3c_32g。
虚拟化后信息如下：
for i in {4..7}; do npu-smi info -t info-vnpu -i $i -c 0; done
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 164 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 165 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 180 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 181 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 196 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 197 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 212 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 213 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
容器启动脚本：
--shm-size 500g \
--device=/dev/vdavinci164:/dev/davinci0 \
--device=/dev/vdavinci165:/dev/davinci1 \
--device=/dev/vdavinci180:/dev/davinci2 \
--device=/dev/vdavinci181:/dev/davinci3 \
--device=/dev/vdavinci196:/dev/davinci4 \
--device=/dev/vdavinci197:/dev/davinci5 \
--device=/dev/vdavinci212:/dev/davinci6 \
--device=/dev/vdavinci213:/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--device /dev/devmm_svm \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin:/usr/local/sbin \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/hccn.conf:/etc/hccn.conf \
-v /etc/vnpu.cfg:/etc/vnpu.cfg \
-v /data/ZhipuAI/GLM-Z1-32B-0414:/model/GLM-Z1-32B-0414 \
3362cc0f4f02 \
bash
./mindieservice_daemon启动报错。
[2025-04-24 17:39:30,750] [370] [281464203833696] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
[2025-04-24 17:40:02,502] [347] [281464222380384] [llm] [INFO][logging.py-227] : model:
FlashChatglmForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(embedding): Embedding(
(word_embeddings): TensorEmbedding()
)
(encoder): GLMTransformer(
(layers): ModuleList(
(0-60): 61 x FlashChatglmLayer(
(self_attn): FlashChatglmAttention(
(o_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
(query_key_value): TensorParallelColumnLinear(
(linear): FastLinear()
)
)
(post_self_attn_layernorm): ChatglmRMSNorm()
(mlp): ChatglmMLP(
(gate_up_proj): TensorParallelColumnLinear(
(linear): FastLinear()
)
(down_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(post_mlp_layernorm): ChatglmRMSNorm()
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
)
(final_layernorm): RMSNorm()
)
(output_layer): TensorParallelHead(
(linear): FastLinear()
)
(rotary_pos_emb): RotaryEmbedding()
)
[2025-04-24 17:40:03,252] [347] [281464222380384] [llm] [INFO][logging.py-227] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 128, 1, 128])
[2025-04-24 17:40:03,253] [347] [281464222380384] [llm] [INFO][logging.py-227] : >>>>>>id of kcache is 281472858727408 id of vcache is 281472858727888
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.564 [api_c_device.cc:406]1997 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.590 [api_c_device.cc:406]1994 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.609 [error_message_manage.cc:53]1997 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(349,mindie_llm_backend_connector):2025-04-24-17:40:18.518.731 [api_c_device.cc:406]2006 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(391,mindie_llm_backend_connector):2025-04-24-17:40:18.518.630 [api_c_device.cc:406]2004 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.790 [error_message_manage.cc:53]1994 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.812 [error_message_manage.cc:53]1997 FuncErrorReason:[INIT][DEFAULT]rtGetPairDevicesInfo execute failed, reason=[driver error:feature not support]
[ERROR] RUNTIME(370,mindie_llm_backend_connector):2025-04-24-17:40:18.518.763 [api_c_device.cc:406]2015 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(384,mindie_llm_backend_connector):2025-04-24-17:40:18.518.771 [api_c_device.cc:406]2011 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(391,mindie_llm_backend_connector):2025-04-24-17:40:18.518.856 [error_message_manage.cc:53]2004 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.861 [error_message_manage.cc:53]1994 FuncErrorReason:[INIT][DEFAULT]rtGetPairDevicesInfo execute failed, reason=[driver error:feature not support]
[ERROR] RUNTIME(349,mindie_llm_backend_connector):2025-04-24-17:40:18.518.843 [error_message_manage.cc:53]2006 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(370,mindie_llm_backend_connector):2025-04-24-17:40:18.518.910 [error_message_manage.cc:53]2015 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(384,mindie_llm_backend_connector):2025-04-24-17:40:18.518.923 [error_message_manage.cc:53]2011 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
这是什么问题？
点赞
收藏
回复
分享
举报
已解决
我把4张910B3做了vnpu，虚拟化实例模板为vir10_3c_32g。
虚拟化后信息如下：
for i in {4..7}; do npu-smi info -t info-vnpu -i $i -c 0; done
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 164 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 165 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 180 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 181 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 196 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 197 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
+-------------------------------------------------------------------------------+
| NPU resource static info as follow: |
| Format:Free/Total NA: Currently, query is not supported. |
| AICORE Memory AICPU VPC VENC VDEC JPEGD JPEGE PNGD |
| GB |
|===============================================================================|
| 0/20 0/64 1/7 1/9 0/0 0/2 0/24 0/4 NA/NA |
+-------------------------------------------------------------------------------+
| Total number of vnpu: 2 |
+-------------------------------------------------------------------------------+
| Vnpu ID | Vgroup ID | Container ID | Status | Template Name |
+-------------------------------------------------------------------------------+
| 212 | 0 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
| 213 | 1 | 000000000000 | 0 | vir10_3c_32g |
+-------------------------------------------------------------------------------+
容器启动脚本：
--shm-size 500g \
--device=/dev/vdavinci164:/dev/davinci0 \
--device=/dev/vdavinci165:/dev/davinci1 \
--device=/dev/vdavinci180:/dev/davinci2 \
--device=/dev/vdavinci181:/dev/davinci3 \
--device=/dev/vdavinci196:/dev/davinci4 \
--device=/dev/vdavinci197:/dev/davinci5 \
--device=/dev/vdavinci212:/dev/davinci6 \
--device=/dev/vdavinci213:/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--device /dev/devmm_svm \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin:/usr/local/sbin \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/hccn.conf:/etc/hccn.conf \
-v /etc/vnpu.cfg:/etc/vnpu.cfg \
-v /data/ZhipuAI/GLM-Z1-32B-0414:/model/GLM-Z1-32B-0414 \
3362cc0f4f02 \
bash
./mindieservice_daemon启动报错。
[2025-04-24 17:39:30,750] [370] [281464203833696] [llm] [INFO][dist.py-81] : initialize_distributed has been Set
[2025-04-24 17:40:02,502] [347] [281464222380384] [llm] [INFO][logging.py-227] : model:
FlashChatglmForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(embedding): Embedding(
(word_embeddings): TensorEmbedding()
)
(encoder): GLMTransformer(
(layers): ModuleList(
(0-60): 61 x FlashChatglmLayer(
(self_attn): FlashChatglmAttention(
(o_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
(query_key_value): TensorParallelColumnLinear(
(linear): FastLinear()
)
)
(post_self_attn_layernorm): ChatglmRMSNorm()
(mlp): ChatglmMLP(
(gate_up_proj): TensorParallelColumnLinear(
(linear): FastLinear()
)
(down_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(post_mlp_layernorm): ChatglmRMSNorm()
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
)
(final_layernorm): RMSNorm()
)
(output_layer): TensorParallelHead(
(linear): FastLinear()
)
(rotary_pos_emb): RotaryEmbedding()
)
[2025-04-24 17:40:03,252] [347] [281464222380384] [llm] [INFO][logging.py-227] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 128, 1, 128])
[2025-04-24 17:40:03,253] [347] [281464222380384] [llm] [INFO][logging.py-227] : >>>>>>id of kcache is 281472858727408 id of vcache is 281472858727888
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.564 [api_c_device.cc:406]1997 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.590 [api_c_device.cc:406]1994 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.609 [error_message_manage.cc:53]1997 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(349,mindie_llm_backend_connector):2025-04-24-17:40:18.518.731 [api_c_device.cc:406]2006 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(391,mindie_llm_backend_connector):2025-04-24-17:40:18.518.630 [api_c_device.cc:406]2004 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.790 [error_message_manage.cc:53]1994 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(356,mindie_llm_backend_connector):2025-04-24-17:40:18.518.812 [error_message_manage.cc:53]1997 FuncErrorReason:[INIT][DEFAULT]rtGetPairDevicesInfo execute failed, reason=[driver error:feature not support]
[ERROR] RUNTIME(370,mindie_llm_backend_connector):2025-04-24-17:40:18.518.763 [api_c_device.cc:406]2015 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(384,mindie_llm_backend_connector):2025-04-24-17:40:18.518.771 [api_c_device.cc:406]2011 rtGetPairDevicesInfo:[INIT][DEFAULT]ErrCode=207000, desc=[driver error:feature not support], InnerCode=0x702001d
[ERROR] RUNTIME(391,mindie_llm_backend_connector):2025-04-24-17:40:18.518.856 [error_message_manage.cc:53]2004 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(377,mindie_llm_backend_connector):2025-04-24-17:40:18.518.861 [error_message_manage.cc:53]1994 FuncErrorReason:[INIT][DEFAULT]rtGetPairDevicesInfo execute failed, reason=[driver error:feature not support]
[ERROR] RUNTIME(349,mindie_llm_backend_connector):2025-04-24-17:40:18.518.843 [error_message_manage.cc:53]2006 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(370,mindie_llm_backend_connector):2025-04-24-17:40:18.518.910 [error_message_manage.cc:53]2015 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(384,mindie_llm_backend_connector):2025-04-24-17:40:18.518.923 [error_message_manage.cc:53]2011 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
这是什么问题？