communication_statistic (Collective Communication Operator Statistics)

The timeline information of the collective communication operator and computation and communication pipeline overlapping data is displayed at the Communication layer in the msprof_*.json file, and the summary information is summarized in the communication_statistic_*.csv file. Additionally, the msprof_*.json file displays the Overlap Analysis for computation and communication pipeline overlapping data analysis.

Collective communication operator data can only be collected and parsed in scenarios where inter-device communications exist, such as the multi-device, multi-server, and cluster scenarios.

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product / Atlas A3 inference product	√
Atlas A2 training product / Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	√
Atlas inference product	√
Atlas training product	√
Atlas 350 Accelerator Card	√

Data at the Communication Layer in msprof_*.json

The following figure shows data at the Communication layer in the msprof_*.json file.

Figure 1 Information about large communication operators

Figure 2 Information about small communication operators

In multi-rank, multi-node, or cluster scenarios, ranks communicate with each other to form communicators. The Communication layer collects the time consumptions of communication operators according to the arranged communicators. In this file, you can find the communication operator that takes the longest time.

**Table 1** Field description
Field	Description
Public information
Group * Communication (communicator name, determined by the reported name)	Communication operators in a communicator. A device (rank) may exist in different communicators, and a group identifies the behavior of the current device in the current communicator.
Plane ID	Network plane ID. In terms of parallel scheduling and execution of multiple transmit and receive communication links, each plane is a concurrent communication dimension.
Title	API name of a component.
Start	Start point on the timeline, which is automatically aligned with that in chrome trace (ms).
Wall Duration	Time taken by the calls to an API (ms).
Self Time	Execution time of the current instruction, in ms.
Information about large communication operators
connection_id	ID of the connection between a CANN API and an NPU operator when the former is delivered to the latter.
model id	Model ID.
data_type	Data type.
alg_type	Algorithm type in each phase of communication operators, which can be MESH, RING, NB, HD, NHR, PIPELINE, PAIRWISE, or STAR.
count	Data transmission count.
relay	Whether link replay happens to the communication operator: yes (link relay occurs) or no (no link relay occurs). Applicability: Atlas A2 training product / Atlas A2 inference product : Only no is displayed, with no specific meaning. Atlas A3 training product / Atlas A3 inference product
retry	Whether execution retry happens to the communication operator: yes (execution retry occurs) or no (no execution retry occurs). Applicability: Atlas A2 training product / Atlas A2 inference product Atlas A3 training product / Atlas A3 inference product
Information about small communication operators
notify id	The unique notify ID. The notify ID is valid only for notify tasks and RDMA send tasks used to transmit the notify record signals. For other task types, the notify ID is invalid and is displayed as 18446744073709551615.
duration estimated(us)	Estimated task duration, in μs.
stream id	Stream ID.
task id	Task ID.
task type	Task type.
src rank	Source rank.
dst rank	Destination rank. The value 4294967295 indicates a local on-chip operation.
transport type	Transmission type, such as LOCAL, SDMA, or RDMA.
size(Byte)	Data volume, in bytes. This field is invalid when the task type is notify and will be set to 0.
data type	Data type.
link type	Link type, which can be HCCS, PCIe, RoCE.
bandwidth(GB/s)	Bandwidth, in GB/s.
model id	Model ID.

Pipeline Overlapping Analysis of Computation and Communication

Overlap Analysis in msprof_*.json is the pipeline overlapping analysis data of computation and communication, which is controlled by task-time and --hccl. See Figure 3.

Computation and communication are sometimes parallel. You can check the pipeline overlapping time (time when computation and communication are parallel) to determine the computation and communication efficiencies.

Figure 3 Effect of pipeline overlapping of computation and communication

**Table 2** Field description
Field	Description
Communication	Communication time. This field is not displayed in the single-device scenario because no communication is involved.
Communication(Not Overlapped)	Communication time that is not overlapped. This field is not displayed in the single-device scenario because no communication is involved.
Computing	Computation time
Free	Interval.
Start	Time when the current API starts to be called, in ms.
Wall Duration	Time taken by the calls to an API, in ms.

communication_statistic_*.csv File

The following shows the file content format.

Figure 4 communication_statistic_*.csv

communication_statistic_*.csv contains the collective communication operator statistics, through which you can learn the time consumption of an operator type and the time consumption ratio of each communication operator in collective communication to determine whether the operator can be optimized.

**Table 3** Field description
Field	Description
Device_id	Device ID.
OP Type	Type of a collective communication operator.
Count	Number of times that collective communication operators are executed.
Total Time(us)	Total execution time of collective communication operators, in μs.
Min Time(us)	Minimum execution time of collective communication operators, in μs.
Avg Time(us)	Average execution time of collective communication operators, μs.
Max Time(us)	Maximum execution time of collective communication operators, in μs.
Ratio(%)	Ratio of execution time of collective communication operators to the overall collective communication time.

Parent topic: Profile Data File References