Prometheus Metrics API

Function Description

Provides the Metrics API for Prometheus to call and integrate.

URL

GET http://ip:port/metrics

For security purposes, NPU Exporter enables the container-level port (8082 by default) by default. The request IP address is the IP address of the Kubernetes container. If the Kubernetes network plugin is Calico, the network policy is set to one that allows the access of the application with the label app=prometheus.

Request Parameters

None

Response Description

Data is returned in the dedicated format of Prometheus. The following information is for reference only. For details about the data, see the following content or Data Description. You can ignore the data provided by Prometheus. Data supported by a product may vary.

...

# HELP machine_npu_nums Amount of npu installed on the machine.
# TYPE machine_npu_nums gauge
machine_npu_nums 8
# HELP npu_chip_info_aicore_current_freq the npu ai core current frequency, unit is 'MHz'
# TYPE npu_chip_info_aicore_current_freq gauge
npu_chip_info_aicore_current_freq{container_name="",id="0",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:61:00.0",pod_name="",vdie_id="185011D4-21104518-A0C4ED94-14CC040A-56102003"} 1000 1723621883587
npu_chip_info_aicore_current_freq{container_name="",id="1",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:DB:00.0",pod_name="",vdie_id="185011D4-21E04718-93B2ED94-14CC040A-BF102003"} 1000 1723621883932
npu_chip_info_aicore_current_freq{container_name="",id="2",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:B2:00.0",pod_name="",vdie_id="185011D4-20C02418-59D4ED94-14CC040A-F9102003"} 1000 1723621884277
npu_chip_info_aicore_current_freq{container_name="",id="3",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:3E:00.0",pod_name="",vdie_id="185011D4-21502C18-0464ED94-14CC040A-6E102003"} 1000 1723621884682
npu_chip_info_aicore_current_freq{container_name="",id="4",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:60:00.0",pod_name="",vdie_id="185011D4-21A02418-64946D94-14CC040A-F8102003"} 1000 1723621885026
npu_chip_info_aicore_current_freq{container_name="",id="5",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:DA:00.0",pod_name="",vdie_id="185011DC-21F02B18-C4B66D94-14CC040A-57102003"} 1000 1723621885385
npu_chip_info_aicore_current_freq{container_name="",id="6",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:B1:00.0",pod_name="",vdie_id="185011D4-20602118-14646D94-14CC040A-8A102003"} 1000 1723621885784
npu_chip_info_aicore_current_freq{container_name="",id="7",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:3D:00.0",pod_name="",vdie_id="185011D4-21504C18-10656D94-14CC040A-6B102003"} 1000 1723621886131
# HELP npu_chip_info_bandwidth_rx the npu interface receive speed, unit is 'MB/s'
# TYPE npu_chip_info_bandwidth_rx gauge
npu_chip_info_bandwidth_rx{container_name="",id="0",model_name="910A-Ascend-V1",namespace="",pcie_bus_info="0000:61:00.0",pod_name="",vdie_id="185011D4-21104518-A0C4ED94-14CC040A-56102003"} 0 1723621883587
...

This API can be used to query both the default and custom metrics groups. Custom Metric Development describes how to customize a metrics group, and the default metrics group contains the following parts. The collection and reporting of a metrics group are governed by its corresponding configuration parameter. When enabled, the metrics group is collected and reported; when disabled, it is neither collected nor reported.

Version Information
NPU Information
vNPU Information
Network Information
DDR Information
On-chip Memory Information
HCCS Information
PCIe Information
RoCE Information
SIO Information
Optical Module Information

For details, see Label Description.

If the process is running on a host and the pod does not use NPUs, the values of pod_name, container_name, and namespace are empty.
NPU Exporter obtains data information by calling the underlying HDK APIs. For more details, see HDK APIs to Be Called.
NPU Exporter does not report data if it does not support the listed products or HDK APIs fail to be called when a data record is queried.

Version Information

**Table 1** Version information
Type	Data Name	Data Description	Data Label	Field Type	Value/Unit	Supported Product
Version	npu_exporter_version_info	NPU Exporter version information	exporterVersion: current NPU Exporter version	String	1: placeholder character, with no specific sense	Atlas training products Atlas A2 training products Atlas A3 training products Inference server (with Atlas 300I inference cards) Atlas inference products Atlas 800I A2 inference server A200I A2 Box heterogeneous component

NPU Information

**Table 2** NPU information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
NPU	npu_chip_info_overall_utilization	Overall usage of the Ascend AI processor	Label 1	Unit: %	Atlas A2 training products Atlas A3 training products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
NPU	npu_chip_info_vector_utilization	AI Vector usage of the Ascend AI processor	Label 1	Unit: %	Atlas A2 training products Atlas A3 training products Inference server (with Atlas 300I inference cards) Atlas inference products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
NPU	machine_npu_nums	Number of Ascend AI processors	-	Unit: Count	Atlas training products Atlas A2 training products Atlas A3 training products Inference server (with Atlas 300I inference cards) Atlas inference products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
NPU	npu_chip_info_voltage	Voltage of the Ascend AI processor	Label 1	Unit: V
NPU	First error code: npu_chip_info_error_code Other error codes: npu_chip_info_error_code_X	Error code of the Ascend AI processor This field is not reported when Ascend AI processors do not have error codes. NOTE: For Prometheus: If an Ascend AI processor contains multiple error codes, only the first 10 error codes can be reported due to Prometheus format restrictions. The value of X ranges from 1 to 9. Telegraf: A maximum of 128 error codes can be reported. For details about the error codes, see Processor Fault Code Reference Documents.	Label 1	-
NPU	npu_chip_info_name	Name and ID of the Ascend AI processor	Label 1	-
NPU	npu_chip_info_health_status	Health status of the Ascend AI processor	Label 1	The value can be 0 or 1. 1: healthy 0: unhealthy
NPU	npu_chip_info_temperature	Temperature of the Ascend AI processor	Label 1	Unit: °C
NPU	npu_chip_info_process_info	Information about the process that occupies the Ascend AI processor Prometheus: The value is the memory used by the process. Telegraf: The value 0 is reported only when no process occupies the Ascend AI processor.	Label 2	Unit: MB
NPU	npu_chip_info_process_info_num	Number of processes that occupy the Ascend AI processor	Label 1	Unit: integer
NPU	npu_container_info	NPU container information NOTE: Telegraf does not support reporting of this field.	Label 5	-
NPU	npu_chip_info_power	Power consumption of the Ascend AI processor NOTE: For Atlas inference products, this field indicates card power consumption. For other products, this field indicates Ascend AI Processor power consumption.	Label 1	Unit: watt (W)
NPU	npu_chip_info_utilization	AI Core usage of the Ascend AI processor	Label 1	Unit: %
NPU	npu_chip_info_aicore_current_freq	Current frequency of the AI Core of the Ascend AI processor	Label 1	Unit: MHz
NPU	container_npu_utilization	AI Core usage of the NPU with container information NOTE: Telegraf does not support reporting of this field.	Label 1	Unit: %
NPU	npu_chip_info_serial_number	SN of the Ascend AI processor	Label 6	1: placeholder character, with no specific sense
NPU	npu_chip_info_network_status	Network health status of the Ascend AI processor	Label 1	The value can be 0 or 1. 1: The network is healthy and can be connected. 0: The network is unhealthy and cannot be connected.	Atlas training products Atlas A2 training products Atlas A3 training products A200I A2 Box heterogeneous component Atlas 800I A2 inference server
NPU	container_npu_total_memory	Total memory size of the NPU with container information NOTE: Telegraf does not support reporting of this field.	Label 1	Unit: MB
NPU	container_npu_used_memory	Used memory of the NPU with container information NOTE: Telegraf does not support reporting of this field.	Label 1	Unit: MB

vNPU Information

**Table 3** vNPU information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
vNPU	vnpu_pod_aicore_utilization	AI Core usage of a vNPU	Label 3	Unit: %	Atlas inference product
vNPU	vnpu_pod_total_memory	Total memory of a vNPU	Label 3	Unit: KB
vNPU	vnpu_pod_used_memory	Memory used by a vNPU	Label 3	Unit: KB

Network Information

Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
Network	npu_chip_info_bandwidth_rx	Real-time RX rate of the network port of the Ascend AI processor	Label 1	Unit: MB/s	Atlas training products Atlas A2 training products Atlas A3 training products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
Network	npu_chip_info_bandwidth_tx	Real-time TX rate of the network port of the Ascend AI processor	Label 1	Unit: MB/s
Network	npu_chip_info_link_status	Network port link status of the Ascend AI processor	Label 1	The value can be 0 or 1. 1: UP 0: DOWN
Network	npu_chip_link_speed	Default network port rate of the Ascend AI processor.	Label 1	Unit: MB/s
Network	npu_chip_link_up_num	Number of times when the network port of the Ascend AI processor is up.	Label 1	Unit: integer

DDR Information

**Table 4** DDR information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
DDR	npu_chip_info_used_memory	Used DDR memory of the Ascend AI processor	Label 1	Unit: MB	Atlas training products Inference server (with Atlas 300I inference cards) Atlas inference products
DDR	npu_chip_info_total_memory	Total DDR memory of the Ascend AI processor	Label 1	Unit: MB

On-chip Memory Information

**Table 5** On-chip memory information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
On-chip memory	npu_chip_info_hbm_used_memory	Used on-chip memory of the Ascend AI processor	Label 1	Unit: MB	Atlas training products Atlas A2 training products Atlas A3 training products A200I A2 Box heterogeneous component Atlas 800I A2 inference server
On-chip memory	npu_chip_info_hbm_total_memory	Total on-chip memory of the Ascend AI processor	Label 1	Unit: MB
On-chip memory	npu_chip_info_hbm_ecc_enable_flag	ECC enabling status of the on-chip memory of the Ascend AI processor	Label 1	The value can be 1 or 0. 0: ECC detection is disabled. 1: ECC detection is enabled.
On-chip memory	npu_chip_info_hbm_ecc_single_bit_error_cnt	Current number of single-bit errors of the on-chip memory of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_ecc_double_bit_error_cnt	Current number of double-bit errors of the on-chip memory of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_ecc_total_single_bit_error_cnt	Total number of single-bit errors during the on-chip memory lifecycle of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_ecc_total_double_bit_error_cnt	Total number of double-bit errors during the on-chip memory lifecycle of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_ecc_single_bit_isolated_pages_cnt	Number of isolated memory pages with single-bit errors of the on-chip memory of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_ecc_double_bit_isolated_pages_cnt	Number of isolated memory pages with double-bit errors of the on-chip memory of the Ascend AI processor	Label 1	-
On-chip memory	npu_chip_info_hbm_utilization	On-chip memory usage of the Ascend AI processor.	Label 1	Unit: %
On-chip memory	npu_chip_info_hbm_temperature	On-chip memory temperature of the Ascend AI processor	Label 1	Unit: °C
On-chip memory	npu_chip_info_hbm_bandwidth_utilization	On-chip memory bandwidth usage of the Ascend AI processor	Label 1	Unit: %

HCCS Information

**Table 6** HCCS information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
HCCS	npu_chip_info_hccs_statistic_info_tx_cnt_X X range: 1 to 7 (Atlas A2 training products or Atlas 900 A3 SuperPoD) or 2 to 7 (Atlas 9000 A3 SuperPoD) .	Number of packets transmitted over the Xth HDLC link. The unit is flit. If the collection fails, -1 is reported.	Label 1	-	Atlas A2 training products Atlas A3 training products
HCCS	npu_chip_info_hccs_statistic_info_rx_cnt_X X range: 1 to 7 (Atlas A2 training products or Atlas 900 A3 SuperPoD)or 2 to 7 (Atlas 9000 A3 SuperPoD)	Number of packets received over the Xth HDLC link. The unit is flit. If the collection fails, -1 is reported.	Label 1	-
HCCS	npu_chip_info_hccs_statistic_info_crc_err_cnt_X X range: 1 to 7 (Atlas A2 training products or Atlas 900 A3 SuperPoD ) or 2 to 7 (Atlas 9000 A3 SuperPoD).	Number of packets with CRC errors received over the Xth HDLC link. The unit is flit. If the collection fails, -1 is reported.	Label 1	-
HCCS	npu_chip_info_hccs_bandwidth_info_profiling_time	Sampling duration of the HCCS link bandwidth. The value ranges from 1 to 1000.	Label 1	Unit: ms
HCCS	npu_chip_info_hccs_bandwidth_info_total_tx	Total bandwidth for sending data on HCCS links. -1 is reported when data collection fails.	Label 1	Unit: GB/s
HCCS	npu_chip_info_hccs_bandwidth_info_total_rx	Total bandwidth for receiving data on HCCS links. -1 is reported when data collection fails.	Label 1	Unit: GB/s
HCCS	npu_chip_info_hccs_bandwidth_info_tx_X X range: 1 to 7 (Atlas A2 training products and Atlas 900 A3 SuperPoD ) or 2 to 7 (Atlas 9000 A3 SuperPoD).	Total bandwidth for sending data on a single HCCS link. -1 is reported when data collection fails.	Label 1	Unit: GB/s
HCCS	npu_chip_info_hccs_bandwidth_info_rx_X X range: 1 to 7 (Atlas A2 training products and Atlas 900 A3 SuperPoD ) or 2 to 7 (Atlas 9000 A3 SuperPoD).	Total bandwidth for receiving data on a single HCCS link. -1 is reported when data collection fails.	Label 1	Unit: GB/s

PCIe Information

**Table 7** PCIe information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
PCIe	npu_chip_info_pcie_rx_p_bw	Remote-write PCIe bandwidth obtained by the Ascend AI processor	Label 4	Unit: MB/ms	Atlas A2 training products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
PCIe	npu_chip_info_pcie_rx_np_bw	Remote-read PCIe bandwidth obtained by the Ascend AI processor	Label 4	Unit: MB/ms
PCIe	npu_chip_info_pcie_rx_cpl_bw	Remote-read PCIe bandwidth with CPL response obtained by the Ascend AI processor	Label 4	Unit: MB/ms
PCIe	npu_chip_info_pcie_tx_p_bw	Remote-write PCIe bandwidth obtained by the Ascend AI processor	Label 4	Unit: MB/ms
PCIe	npu_chip_info_pcie_tx_np_bw	Remote-read PCIe bandwidth obtained by the Ascend AI processor	Label 4	Unit: MB/ms
PCIe	npu_chip_info_pcie_tx_cpl_bw	Remote-write PCIe bandwidth with CPL response obtained by the Ascend AI processor	Label 4	Unit: MB/ms

RoCE Information

**Table 8** RoCE information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
RoCE	npu_chip_mac_rx_pause_num	Total number of pause frames received by the MAC	Label 1	-	Atlas training products Atlas A2 training products Atlas A3 training products Atlas 800I A2 inference server A200I A2 Box heterogeneous component
RoCE	npu_chip_mac_tx_pause_num	Total number of pause frames transmitted by the MAC	Label 1	-
RoCE	npu_chip_mac_rx_pfc_pkt_num	Total number of PFC frames received by the MAC	Label 1	-
RoCE	npu_chip_mac_tx_pfc_pkt_num	Total number of PFC frames transmitted by the MAC	Label 1	-
RoCE	npu_chip_mac_rx_bad_pkt_num	Total number of bad packets received by the MAC	Label 1	-
RoCE	npu_chip_mac_tx_bad_pkt_num	Total number of bad packets transmitted by the MAC	Label 1	-
RoCE	npu_chip_mac_tx_bad_oct_num	Total bytes of bad packets transmitted by the MAC	Label 1	-
RoCE	npu_chip_mac_rx_bad_oct_num	Total bytes of bad packets received by the MAC	Label 1	-
RoCE	npu_chip_roce_rx_all_pkt_num	Total number of packets received by the RoCE	Label 1	-
RoCE	npu_chip_roce_tx_all_pkt_num	Total number of packets transmitted by the RoCE	Label 1	-
RoCE	npu_chip_roce_rx_err_pkt_num	Total number of bad packets received by the RoCE	Label 1	-
RoCE	npu_chip_roce_tx_err_pkt_num	Total number of bad packets transmitted by the RoCE	Label 1	-
RoCE	npu_chip_roce_rx_cnp_pkt_num	Number of CNP packets received by the RoCE	Label 1	-
RoCE	npu_chip_roce_tx_cnp_pkt_num	Number of CNP packets transmitted by the RoCE	Label 1	-
RoCE	npu_chip_roce_new_pkt_rty_num	Number of packet transmission retries by the RoCE	Label 1	-
RoCE	npu_chip_roce_unexpected_ack_num	Number of unexpected ACK packets received by the RoCE. These packets are deprecated by the NPU, without affecting services.	Label 1	-
RoCE	npu_chip_roce_out_of_order_num	Number of PSN packets received by the RoCE > expected PSN packets or duplicate PSN packets. If packets are out of order or lost, transmission retry is triggered.	Label 1	-
RoCE	npu_chip_roce_verification_err_num	Number of packets with field verification failure received by the RoCE. The field verification includes the ICRC, packet length, and destination port number.	Label 1	-
RoCE	npu_chip_roce_qp_status_err_num	Number of packets received by the RoCE during abnormal QP connection	Label 1	-
RoCE	npu_chip_info_rx_ecn_num	Number of explicit congestion notifications (ECNs) received by the Ascend AI processor's network	Label 1	-
RoCE	npu_chip_info_rx_fcs_num	Number of frame check sequences (FCSs) received by the Ascend AI processor's network	Label 1	-

SIO Information

**Table 9** SIO information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
SIO	npu_chip_info_sio_crc_tx_err_cnt	Number of error packets transmitted by the SIO	Label 1	-	Atlas A3 training products
SIO	npu_chip_info_sio_crc_rx_err_cnt	Number of error packets received by the SIO	Label 1	-	Atlas A3 training products

Optical Module Information

**Table 10** Optical module information
Type	Data Name	Data Description	Data Label	Value/Unit	Supported Product
Optical module	npu_chip_optical_state	Optical module status	Label 1	The value can be 0 or 1. 0: not detected 1: detected	Atlas training products Atlas A2 training products Atlas 900 A3 SuperPoD Atlas 800I A2 inference server A200I A2 Box heterogeneous component
Optical module	npu_chip_optical_tx_power_X (X ranges from 0 to 3)	TX power of an optical module	Label 1	Unit: mW
Optical module	npu_chip_optical_rx_power_X (X ranges from 0 to 3)	RX power of an optical module	Label 1	Unit: mW
Optical module	npu_chip_optical_vcc	Optical module voltage	Label 1	Unit: mV
Optical module	npu_chip_optical_temp	Optical module temperature	Label 1	Unit: °C

Label Description

The following table describes labels.

**Table 11** Label information
Label	Field Description	Field Type
Label 1	container_name: container name	String
	id: NPU ID	String
	model_name: Ascend AI processor name	String
	namespace: namespace	String
	pcie_bus_info: PCIe information of the Ascend AI processor	String
	pod_name: pod name	String
	vdie_id: Unique ID of the Ascend AI processor, which can be used as the NPU's UUID.	String
Label 2	container_name: container name. The output format is Pod Namespace_Pod name_Container name. If the process is running on the host, the value is left empty.	String
	container_id: container ID	String
	id: NPU ID	String
	model_name: Ascend AI processor name	String
	namespace: namespace	String
	pcie_bus_info: PCIe information of the Ascend AI processor	String
	pod_name: pod name	String
	vdie_id: Unique ID of the Ascend AI processor, which can be used as the NPU's UUID.	String
	process_id: NPU Exporter is started by the root user in the privileged container. The queried PID is the host PID. In the host scenario, the queried PID is related to the process on the host. In other container scenarios, the queried PID is related to the process in the current container.	String
Label 3	aicore_count: number of vNPU cores	Float
	container_name: container name	String
	id: NPU ID	String
	model_name: Ascend AI processor name	String
	namespace: namespace	String
	pod_name: pod name	String
	v_dev_id: unique identifier of a vNPU	String
	is_virtual: whether the device is a virtual device	Bool
Label 4	container_name: container name	String
	id: NPU ID	String
	model_name: Ascend AI processor name	String
	namespace: namespace	String
	pcie_bus_info: PCIe information of the Ascend AI processor	String
	pod_name: pod name	String
	vdie_id: Unique ID of the Ascend AI processor, which can be used as the NPU's UUID.	String
	pcie_bw_type: statistics on PCIe bandwidth for writing data to the remote end minPcieBw: minimum value maxPcieBw: maximum value avgPcieBw: average value	String
Label 5	container_name: container name	String
	containerName: container name. The output format is Pod Namespace_Pod name_Container name.	String
	containerID: container ID	String
	npuID: NPU ID	String
	model_name: Ascend AI processor name	String
	namespace: namespace	String
	pcie_bus_info: PCIe information of the Ascend AI processor	String
	pod_name: pod name	String
	vdie_id: Unique ID of the Ascend AI processor, which can be used as the NPU's UUID.	String
Label 6	container_name: container name	String
	id: NPU ID	String
	model_name: Ascend AI Processor name	String
	namespace: namespace	String
	pcie_bus_info: PCIe information of the Ascend AI Processor	String
	pod_name: pod name	String
	serial_number: SN of the Ascend AI Processor. If this parameter is not set, NA is reported.	String
	vdie_id: Unique ID of the Ascend AI Processor, which can be used as the NPU's UUID.	String

HDK APIs to Be Called

NPU Exporter obtains data information by calling the underlying HDK APIs. For details about HDK APIs called to obtain the above data, HDK APIs Called by NPU Exporter. To search for the required HDK APIs, perform the following steps:

Visit Ascend Computing Documentation and select a product name to go to the document page. For example, if you are a user of Atlas 800I A2 inference servers, click Atlas 800I A2.
In the navigation pane on the left, click Developer Documents.
In the search box on the home page of the document, search for an API name or keyword to view the API information.

Status Codes

**Table 12** Status codes
Status Code	Description
200	Normal
307	Temporary redirection
500	Internal server error

Parent topic: NPU Exporter