No NPU Is Mounted to a Job Container

Symptom

The following command is run in a job container, but no command output is displayed. This indicates that no NPU device is mounted to the container.

ls /dev/davinci*

Cause Analysis

The default value of useAscendDocker of Ascend Device Plugin is true, indicating that it must be used together with Ascend Docker Runtime. In the following scenarios, Ascend Device Plugin may fail to work with Ascend Docker Runtime:

  • Cause 1: Ascend Docker Runtime is not installed in the environment.
  • Cause 2: Ascend Docker Runtime has been installed, but the Docker service is not restarted.

Solution

For cause 1, install Ascend Docker Runtime by referring to Ascend Docker Runtime, restart the Docker service, delete the old job, and deliver the job again.

For cause 2, restart the Docker service, delete the old job, and deliver the job again.

Ascend Docker Runtime can proactively mount a specified NPU to a container. You can run the following command to query the Docker configuration:

docker info 2>&1 | grep "Default Runtime"

If "ascend" is displayed in the command output, Docker uses Ascend Docker Runtime. The following is an example:

Default Runtime: ascend