Cross-Node P2P Bandwidth Test

Function

The cross-node P2P bandwidth test is primarily used to evaluate the following:

Atlas 850E/Atlas 650E server: cross-node P2P transmission rates and total elapsed time in UBoE (UnifiedBus over Ethernet) networking scenarios
Atlas 950 SuperPoD: cross-chassis P2P transmission rates and total elapsed time in UB CLOS networking scenarios

Scenario

To specify the link to be tested, set the environment variable ADMI_TMP_P2P_LINK_TYPE. For more details, see Environment Variables.
Optional values of ADMI_TMP_P2P_LINK_TYPE:
- Atlas 850E/Atlas 650E server: CLOS8
- Atlas 950 SuperPoD: CLOS2, CLOS6
If ADMI_TMP_P2P_LINK_TYPE is not specified, the Atlas 850E/Atlas 650E server uses the UBoE port by default, and the Atlas 950 SuperPoD uses the CLOS2 port by default.

Test Procedure

The bandwidth can be tested using MPI. If MPI is not installed in the environment, install and configure it in advance by referring to Installing and Configuring the MPI in CANN HCCL Performance Tester, and configure the environment variables by referring to Compilation in CANN HCCL Performance Tester. Only Open MPI 4.1.5 is supported. Do not use parameters, configurations, and environment variables that are not mentioned in the description and examples. For details about other Open MPI execution parameters, environment variables, and configuration files, see Open MPI documentation.

Ensure that the entered IP address is valid.

Parameters

You can run either of the following commands to list the parameters of the bandwidth test command:

ascend-dmi --bw -h

ascend-dmi --bw --help

Table 1 lists only test-specific parameters. For details about other common parameters, see Common Parameters.

**Table 1** Parameters
Parameter	Description	Restrictions	Mandatory
[-bw, --bw, --bandwidth]	Measures the chip bandwidth. -bw is supported, but --bw or --bandwidth is recommended.	-	Yes
[-t, --type]	Specifies the data flows to be tested.	Currently, only the P2P mode is supported.	Yes
[-sp, --sp, --super-pod]	Specifies a cross-node test. The value 2 indicates that the cross-node P2P test is performed in MPI mode.	If the value is 2, this parameter must be specified together with [-dd, --dd, --device-dst] and [-ds, --ds, --device-src].	Yes
[-ds, --ds, --device-src]	Specifies the ID of the source device for a P2P test. This parameter must be specified together with [-dd, --dd, --device-dst].	After [-ds, --ds, --device-src] and [-dd, --dd, --device-dst] are specified, [-d, --device] cannot be specified.	Yes
[-dd, --dd, --device-dst]	Specifies the ID of the destination device for a P2P test. This parameter must be specified together with [-ds, --ds, --device-src].	After [-ds, --ds, --device-src] and [-dd, --dd, --device-dst] are specified, [-d, --device] cannot be specified.	Yes
[-s, --size]	Specifies the size of the data to be transmitted and the display mode of the test result. The size ranges from 1 byte to 4 GB. The default value is 536870912 bytes.	After -s is specified, a number must follow; otherwise, the format is incorrect.	No
[-et, --et, --execute-times]	Specifies the number of iterations, that is, number of copy times in the memory.	The value range is [1, 100000000]. If this parameter is not specified, the default value 40 is used. The values of the -et parameter specified on the two nodes where the cross-node P2P bandwidth test is performed must be the same.	No

Example

Run the mpirun -mca btl_tcp_if_include xx.xx.xx.x2(Source node IP)/port -mca btl_tcp_port_min_v4 32768 -mca btl_tcp_port_min_v6 32768 -host xx.xx.xx.x2(Source node IP):1,xx.xx.xx.x3(Peer node IP):1 -n 2 ascend-dmi --bw -t p2p --sp 2 --dd 0 --ds 1 -q command to test the cross-node P2P bandwidth.

Parameter description:
- -mca btl_tcp_if_include specifies the MPI communication standard and communication port.
- -mca btl_tcp_port_min_v4 specifies the minimum TCP port number used by the MPI IPv4 standard.
- -mca btl_tcp_port_min_v6 specifies the minimum TCP port number used by the MPI IPv6 standard.

No authentication mechanism is available for the ports used by MPI. To prevent malicious connection attacks, perform security hardening by referring to Avoiding MPI All-Zero Listening.

[****@****]# mpirun -mca btl_tcp_if_include xx.xx.xx.x2/22 -mca btl_tcp_port_min_v4 32768 -mca btl_tcp_port_min_v6 32768 -host xx.xx.xx.x2:1,xx.xx.xx.x3:1 -n 2 ascend-dmi --bw -t p2p --sp 2 --dd 0 --ds 1 -q
Unidirectional Peer to Peer Test
Pod: rank 0 device id: 1 to Pod: rank 1 device id: 0
----------------------------------------------------------------
  Size(Bytes)  Execute Times  Bandwidth(GB/s)  Elapsed Time(us)
----------------------------------------------------------------
   536870912         40          328.874329       3264.90
----------------------------------------------------------------

Bidirectional Peer to Peer Test
Pod: rank 0 device id: 1 and Pod: rank 1 device id: 0
----------------------------------------------------------------
  Size(Bytes)  Execute Times  Bandwidth(GB/s)  Elapsed Time(us)
----------------------------------------------------------------
   536870912         40          583.744108       3678.81
----------------------------------------------------------------

The following table describes the parameters in the command output.

**Table 2** Parameters in the command output
Parameter	Description
Unidirectional Peer to Peer Test	Unidirectional P2P
Bidirectional Peer to Peer Test	Bidirectional P2P
Size(Bytes)	Size of data to be transmitted, in byte
Execute Times	Number of iterations
Bandwidth(GB/s)	Chip bandwidth, in GB/s.
Elapsed Time(us)	Execution duration

Parent topic: Performance Test