问题
在Atlas 300I Duo 上用K8S集成docker发布MindIE模型推理任务，describe pod回显：
Warning UnexpectedAdmissionError 17m kubelet Allocate failed due to rpc error: code = Unknown desc = not get valid pod, which is unexpected
各部分版本：
MindIE：1.0.0 | npu driver:24.0.1 | CANN 8.0.0 | Ascend Docker Runtime 6.0.0 | Ascend Device Plugin 6.0.0 | volcano 1.7.0 | Ascend Operator 6.0.0| kubelet kubectl kubeadm 1.23.2
deployment yaml
组件日志
Ascend Device Plugin
Volcano scheduler
Volcano-controller
Ascend-Operator
集群pod一览
点赞
收藏
回复
分享
举报
已解决
问题
在Atlas 300I Duo 上用K8S集成docker发布MindIE模型推理任务，describe pod回显：
Warning UnexpectedAdmissionError 17m kubelet Allocate failed due to rpc error: code = Unknown desc = not get valid pod, which is unexpected
各部分版本：
MindIE：1.0.0 | npu driver:24.0.1 | CANN 8.0.0 | Ascend Docker Runtime 6.0.0 | Ascend Device Plugin 6.0.0 | volcano 1.7.0 | Ascend Operator 6.0.0| kubelet kubectl kubeadm 1.23.2
deployment yaml
labels: app: qwen-7b-onecard spec: nodeName: master-118 schedulerName: volcano containers: - name: qwen-7b-onecard image: mindie:1.0.0-300I-Duo-py311-openeuler24-onecard imagePullPolicy: IfNotPresent ports: - containerPort: 8080 volumeMounts: - name: model mountPath: /mnt/models resources: requests: huawei.com/Ascend310P: 1 limits: huawei.com/Ascend310P: 1 volumes: - name: model hostPath: path: /opt/bcc/storage2/5/模型仓库/DeepSeek-R1-Distill-Qwen-7B type: DirectoryOrCreate组件日志
Ascend Device Plugin
[INFO] 2025/03/15 16:00:55.487724 32998 server/plugin.go:800 request: []string{"Ascend310P-0"} [WARN] 2025/03/15 16:00:55.488872 32998 server/plugin.go:711 no pod passed the filter, request device: [Ascend310P-0], retry: 0 [WARN] 2025/03/15 16:00:56.490498 32998 server/plugin.go:711 no pod passed the filter, request device: [Ascend310P-0], retry: 1 [WARN] 2025/03/15 16:00:57.500828 32998 server/plugin.go:711 no pod passed the filter, request device: [Ascend310P-0], retry: 2 [ERROR] 2025/03/15 16:00:58.501357 32998 server/plugin.go:808 not get valid pod [INFO] 2025/03/15 16:00:58.522327 65 kubeclient/kube_cache.go:117 pod(default/qwen-7b-onecard-5679bb6895-nhwn6) is update to cache [INFO] 2025/03/15 16:00:58.529361 65 kubeclient/kube_cache.go:117 pod(default/qwen-7b-onecard-5679bb6895-nhwn6) is update to cache [INFO] 2025/03/15 16:00:58.529535 65 kubeclient/kube_cache.go:117 pod(default/qwen-7b-onecard-5679bb6895-htz8l) is add to cache [INFO] 2025/03/15 16:00:58.536507 65 kubeclient/kube_cache.go:117 pod(default/qwen-7b-onecard-5679bb6895-htz8l) is update to cache [WARN] 2025/03/15 16:00:58.568055 33000 device/ascendcommon.go:577 get device ip failed, err: get device IP address failed, cardID(32768), deviceID(0), error code: -8255 [WARN] 2025/03/15 16:00:58.569229 33000 device/ascendcommon.go:577 get device ip failed, err: get device IP address failed, cardID(32768), deviceID(1), error code: -8255 [INFO] 2025/03/15 16:00:58.569582 33000 server/plugin.go:800 request: []string{"Ascend310P-0"} [WARN] 2025/03/15 16:00:58.571020 33000 server/plugin.go:711 no pod passed the filter, request device: [Ascend310P-0], retry: 0 [WARN] 2025/03/15 16:00:59.573265 33000 server/plugin.go:711 no pod passed the filter, request device: [Ascend310P-0], retry: 1 [INFO] 2025/03/15 16:01:00.487885 65 kubeclient/kube_cache.go:117 pod(default/qwen-7b-onecard-5679bb6895-nhwn6) is update to cacheVolcano scheduler
Volcano-controller
Ascend-Operator
集群pod一览