Before You Start

Resource monitoring mainly includes: (1) vNPU AI Core usage, total vNPU memory, and used vNPU memory (2) real-time NPU resource usage in training or inference jobs, including the usage, temperature, voltage, memory, and allocation status in containers of Ascend AI processors.

Resource monitoring is a fundamental feature that is not specific to either training or inference scenarios, and is also not specific to the Volcano scheduler or other schedulers. This feature must work with Prometheus or Telegraf. If it works with Prometheus, deploy Prometheus first and call related NPU Exporter APIs to monitor resources. If it works with Telegraf, deploy and run Telegraf to monitor resources.

• Prometheus is a comprehensive open-source monitoring solution that features easy management, high efficiency, scalability, and visualization. It works with NPU Exporter to achieve real-time monitoring of each Ascend AI processor's usage, temperature, voltage, memory, and allocation status in containers. It can also monitor the vNPU AI Core usage, total vNPU memory, and used vNPU memory.
• Telegraf collects statistics on the system and services. It occupies a small amount of memory and supports the expansion of other services. It works with NPU Exporter. You can view the reported information about Ascend AI processors in the command output in your environment.