Job Progress Viewing

Procedure

Run the following command to check the pod running status:

kubectl get pod --all-namespaces

Command output:

NAMESPACE        NAME                                       READY   STATUS    RESTARTS   AGE
...
default          resnetinfer1-2-scpr5                      1/1     Running   0          8s
...

Run the following command to view details about the node running the inference job:

kubectl describe node <hostname>

For example:

kubectl describe node ubuntu

Command output for full NPU scheduling:

...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource              Requests     Limits
  --------              --------     ------
  cpu                   4 (2%)       3500m (1%)
  memory                2140Mi (0%)  4040Mi (0%)
  ephemeral-storage     0 (0%)       0 (0%)
  huawei.com/Ascend310P  1            1
Events:
  Type    Reason    Age   From                Message
  ----    ------    ----  ----                -------
  Normal  Starting  36m   kube-proxy, ubuntu  Starting kube-proxy.
...

In the displayed information, find huawei.com/Ascend310P under Allocated resources. The value of this parameter increases after the inference job is executed. The increased number is the number of NPUs used by the inference job.

Command output for static vNPU scheduling:

...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource              Requests     Limits
  --------              --------     ------
  cpu                   4 (2%)       3500m (1%)
  memory                2140Mi (0%)  4040Mi (0%)
  ephemeral-storage     0 (0%)       0 (0%)
  Ascend310P-2c  1            1
Events:
  Type    Reason    Age   From                Message
  ----    ------    ----  ----                -------
  Normal  Starting  36m   kube-proxy, ubuntu  Starting kube-proxy.
...

In the displayed information, find Ascend310P-2c under Allocated resources. The value of this parameter increases after the inference job is executed. The increased number is the number of vNPUs used by the inference job.

For non-mixed insertion mode of Atlas inference product, Ascend310P or Ascend310P-2c is displayed.
For mixed insertion mode of Atlas inference product, Ascend310P-V, Ascend310P-VPro, or Ascend310P-IPro is displayed.

Parent topic: Use on the CLI (Volcano)