GUI Description

Function

In the Ascend heterogeneous compute architecture, MindStudio Insight displays the running details of the host and device during training or inference in the timeline view, displays the API time consumption on the host and the task time consumption on the device, and associates the host with the device. This helps you quickly identify host or device bottlenecks. In addition, the tool provides various filtering categories and advice to support in-depth tuning.

GUI Display

The Timeline tab page consists of the toolbar (area 1), timeline tree chart (area 2), graphical pane (area 3), and data pane (area 4), as shown in Figure 1.

Figure 1 Timeline tab page

Area 1: toolbar, which contains common shortcut keys. From left to right, the shortcut keys are Marker List, Filter (card or unit), Search, Flow Events, Reset (page restoration), Timeline Zoom Out, and Timeline Zoom In.
Area 2: timeline tree chart. The display in the TEXT scenario is different from that in the DB scenario. For details about the unit information, see Table 1.
- TEXT scenario: displays the layer information of each device in the cluster scenario. The layer information is displayed by rank. The first layer is the rank ID, the second layer is the process or special layer, and the third layer is the thread name. The second layer includes the Python layer data (time consumption information of PyTorch and dotting data), CANN layer data (time consumption data of AscendCL, GE, and Runtime components), bottom-layer NPU data (time consumption data and step trace data of each stream task flow under Ascend Hardware, communication data of Communication and Overlap Analysis, memory data, and other Ascend AI Processor system data), AI Core Freq, and other layers. The layer content varies with the imported data.
- DB scenario: displays information about each host. The first layer is the host name, and the second layer is Host and Rank ID. The Host level displays PyTorch and CANN data by process and thread. The Rank ID level includes the bottom-layer NPU data (including time consumption data and step trace data of each stream task flow under Ascend Hardware, communication data of Communication and Overlap Analysis, memory data, and other Ascend AI Processor system data), AI Core Freq, and other layers. The content of the level under the card varies with the imported data.
Area 3: graphical pane, which displays data of a step. The graphical pane corresponds to the timeline tree chart, which graphically displays the timeline row by row, including the execution sequence and duration of upper-level application operators, components, and APIs.
Area 4: data pane, which displays statistics or operator details. Slice Detail displays details about a selected operator. Slice List displays the operator list in the selected area of a unit. System View displays the summary information about a type of operators. Find displays the information about the searched operators.

**Table 1** Unit information
Level-1 Unit Name	Level-2 Unit Name	Description
Process	Thread	This unit can be displayed only in DB files. The Thread unit contains the PyTorch, CANN, and MSTX units, which display the time consumed by upper-layer application threads in the PyTorch framework, time consumed by threads in the CANN framework, and dotting information, respectively.
Python	Thread	Application layer data. Each sub-unit Thread contains the time consumed by the upper-layer application thread. The data is collected using PyTorch Profiler or msproftx. This unit can be displayed only in text files.
CANN	Thread	Data at the CANN layer. Each sub-unit Thread contains the time consumption data of AscendCL, GE, Runtime, and nodes (operators). For DB files, the level-2 unit name may contain acl, model, node, hccl, runtime, op, queue, trace, and mstx.
MindSpore	Thread	In MindSpore, the time consumed by each phase of the current thread is displayed.
Scope Layer	Thread	In MindSpore, the time consumed by each network layer of the current thread is displayed.
Python GC	Python GC	In PyTorch, if the GC detection is enabled during profile data collection and a GC event occurs during collection, the GC event is recorded in the collected data and is displayed in the Python GC unit.
Ascend Hardware	Stream <id>	Bottom-layer NPU data, task scheduling data, recording the execution duration of each task on different accelerators and AI Core performance metrics during AI task running.
	Stream <id> MSTX domain <domainid>	MSTX device dotting data of stream < id >.
	Step Trace	Step trace data. This unit is displayed only when the *step_trace_.json** file exists.
HBM	HBM <id>/Read	HBM read rate, in MB/s.
HBM	HBM <id>/Write	HBM write rate, in MB/s.
DDR	Read	DDR memory read rate.
DDR	Write	DDR memory write rate.
LLC	LLC <id> Read/Hit Rate LLC <id> Write/Hit Rate	L3 cache read/write speed data, L3 cache read and write throughputs.
LLC	LLC <id> Read/Throughput LLC <id> Write/Throughput	L3 cache read and write hit rates.
NPU_MEM	APP/DDR	Process-level DDR memory usage, in KB.
	APP/HBM	Process-level HBM memory usage, in KB.
	APP/MEMORY	Sum of process-level DDR and HBM memory usages, in KB.
	Device/DDR	Device-level DDR memory usage, in KB.
	Device/HBM	Device-level HBM memory usage, in KB.
	Device/MEMORY	Sum of device-level DDR and HBM memory usages, in KB.
Communication	Group <id> Communication	Communication operators in a communication group. A card (rank) may exist in different communication groups, and a group identifies the behavior of the current device in the current communication group.
Communication	Plane <id>	Information about the collective communication operator. Network plane ID. In terms of parallel scheduling and execution of multiple transmit and receive communication links, each plane is a concurrent communication dimension.
Stars Soc Info	L2 Buffer Bw Level	SoC transmission bandwidth information and L2 buffer bandwidth level information.
Stars Soc Info	Mata Bw Level	Mata bandwidth level information.
acc_pmu	Accelerator {accId}/readBwLevel	Read bandwidth of the DVPP and DSA accelerators.
	Accelerator {accId}/readOstLevel	Concurrent read operations of the DVPP and DSA accelerators.
	Accelerator {accId}/writeBwLevel	Write bandwidth of the DVPP and DSA accelerators.
	Accelerator {accId}/writeOstLevel	Concurrent write operations of the DVPP and DSA accelerators.
Overlap Analysis	Communication	Communication time.
	Communication(Not Overlapped)	Not-overlapped communication time.
	Computing	Computation time.
	Free	Time when the device is neither computing nor communicating. When data is split by step, it is further divided into Preparing and Free. Preparing is used for data preprocessing, loading, and copying.
AI Core Utilization	Average	Average percentage of AI Core instructions. AI Core Utilization units can be displayed only in text files.
AI Core Utilization	Core <id>	Percentage of total execution cycles (counting from the first operator instruction executed by the AI Core to the completion of the last instruction executed) of a task on the AI Core.
AI Core Freq	AI Core Freq	Frequency changes of AI Cores during AI task running. The AI Core Freq unit supports only the profile data exported from the Atlas A2 Training Series Product.
SIO	dat_rx and dat_tx	Rx and Tx bandwidths of the data stream channel. SIO units can be displayed only in text files. SIO units display only the transmission bandwidth between dies in the Atlas A3 Training Series Product.
	req_rx and req_tx	Rx and Tx bandwidths of the request stream channel.
	rsp_rx and rsp_tx	Rx and Tx bandwidths of the response stream channel.
	snp_rx and snp_tx	Rx and Tx bandwidths of the monitor stream channel.
QoS	QoS <id>:OTHERS	Device QoS bandwidth.
NIC	Port <id>/Rx Port <id>/Tx	TEXT scenario: network information on each time node DB scenario: bandwidth information The unit name varies according to the imported data.
RoCE	Port <id>/Rx Port <id>/Tx	RoCE bandwidth. RoCE units can be displayed only in text files.
PCIe	PCIe_cpl	Completion packet that receives write requests, in MB/s. Tx stands for the transmit direction whereas Rx standards for the receive direction.
	PCIe_nonpost	PCIe Non-Posted data transmission bandwidth, in MB/s. Tx stands for the transmit direction whereas Rx standards for the receive direction.
	PCIe_nonpost_latency	Transmission latency in PCIe Non-Posted mode, in μs. Tx stands for the transmit direction whereas Rx standards for the receive direction. PCIe_nonpost_latency does not involve Rx. The value is fixed at 0.
	PCIe_post	PCIe Posted data transmission bandwidth, in MB/s. Tx stands for the transmit direction whereas Rx standards for the receive direction. The unit name varies according to the imported data.
HCCS	txThroughput rxThroughput	Tx and Rx bandwidths (MB/s) of HCCS collective communication.
biu_group	Bandwidth Read	Bandwidth for the BIU bus interface to read instructions. biu_group units can be displayed only in text files.
	Bandwidth Write	Bandwidth for the BIU bus interface to write instructions.
	Latency Read	Latency for the BIU bus interface to read instructions.
	Latency Write	Latency for the BIU bus interface to write instructions.
aic_core_group	Cube	Cycle quantity and ratio of matrix operation instructions in the current sampling period. aic_core_group units can be displayed only in text files.
	Mte1	Cycle quantity and ratio of L1-to-L0A/L0B transfer instructions in the current sampling period.
	Mte2	Cycle quantity and ratio of on-chip memory to AI Core movement instructions in the current sampling period.
	Mte3	Cycle quantity and ratio of AI Core to on-chip memory movement instructions in the current sampling period.
aiv_core_group	Mte1	Cycle quantity and ratio of L1-to-L0A/L0B transfer instructions in the current sampling period. aiv_core_group units can be displayed only in text files.
	Mte2	Cycle quantity and ratio of on-chip memory to AI Core movement instructions in the current sampling period.
	Mte3	Cycle quantity and ratio of AI Core to on-chip memory movement instructions in the current sampling period.
	Scalar	Cycle quantity and ratio of scalar operation instructions in the current sampling period.
	Vector	Cycle quantity and ratio of vector operation instructions in the current sampling period.
Stars Chip Trans	PA Link Rx	Rx level of the PA traffic. When there is HCCS bandwidth, you are not advised referring to this field value, as this field is a coarse-grained statistical value. Stars Chip Trans units can be displayed only in text files.
	PA Link Tx	Tx level of the PA traffic. When there is HCCS bandwidth, you are not advised referring to this field value, as this field is a coarse-grained statistical value.
	PCIE Read Bandwidth	PCIe read bandwidth. When there is PCIe bandwidth, you are not advised referring to this field value, as this field is a coarse-grained statistical value.
	PCIE Write Bandwidth	PCIe write bandwidth. When there is PCIe bandwidth, you are not advised referring to this field value, as this field is a coarse-grained statistical value.
CPU Usage	CPU <id>	Host-side CPU usage.
Memory Usage	Memory Usage	Host-side memory usage.
Disk Usage	Disk Usage	Host-side drive I/O usage.
Network Usage	Network Usage	Host-side network I/O usage.
OS Runtime API	Thread	Host-side syscall and pthreadcall data. OS Runtime API units can be displayed only in text files.

Check the time consumption and interval at each layer in the timeline view to determine whether the corresponding component and operator have performance problems. For example, check whether there is a bottleneck in operator delivery, whether there is a time-consuming kernel, and whether there are redundant conversion operators.

Parent topic: Timeline