Rank Table Configuration (Atlas 350 Accelerator Card)

For the Atlas 350 Accelerator Card, the rank table file must be used alongside the topology file to initialize the HCCL communicator.

The maximum size of a rank table file is 1 GB.
The rank table file is in JSON format. The comments in the JSON file provided in this section are for reference only. Delete the comments when you use it.

Topology File Configuration

The topology file contains the physical topology and routing information. It is automatically configured before delivery and stored in the /usr/local/Ascend/driver/topo/ directory. Learn the format and field description in this file. The following figure shows a network example with two AI servers, each having two NPUs.

Figure 1 Network example

The topology file is in JSON format, with a configuration example as follows:

{
    "version": "2.0",
    "peer_count": 2,
    "peer_list":[
        { "local_id": 0},
        { "local_id": 1}
    ],
    "edge_count": 3,
    "edge_list": [
        {
            "net_layer": 0, // Intra-server connection
            "link_type": "PEER2PEER",
            "protocols": ["UB_CTP"],
            "local_a": 0, // NPU0 in the preceding figure
            "local_a_ports": ["1/0"], // Intra-server port (port 0 on die 1)
            "local_b": 1, // NPU1 in the preceding figure
            "local_b_ports": ["1/0"], // Intra-server port (port 0 on die 1)
            "topo_instance_id": 0,
            "topo_type": "1DMESH",
            "position": "DEVICE"
        },{
            "net_layer": 1, // Inter-server connection
            "link_type": "PEER2NET",
            "protocols": ["UB_CTP"],
            "local_a": 0, // NPU0 in the preceding figure
            "local_a_ports": ["0/4","0/5","0/6","0/7","1/5","1/6"], // Inter-server ports (ports 4, 5, 6, and 7 on die 0 and ports 5 and 6 on die 1)
            "topo_instance_id": 0,
            "topo_type": "CLOS",
            "position": "DEVICE"
        },{
            "net_layer": 1,
            "link_type": "PEER2NET",
            "protocols": ["UB_CTP"],
            "local_a": 1, // Inter-server connection
            "local_a_ports": ["0/4","0/5","0/6","0/7","1/5","1/6"], // Inter-server ports (ports 4, 5, 6, and 7 on die 0 and ports 5 and 6 on die 1)
            "topo_instance_id": 0,
            "topo_type": "CLOS",
            "position": "DEVICE"
        }
    ]
}

**Table 1** Configuration items in the network topology file
Level-1 Configuration Item	Level-2 Configuration Item	Description
version		Mandatory. Version of the topology file template. The value is fixed at 2.0.
peer_count		Mandatory. Number of NPUs on the current AI server. The value ranges from 1 to 65.
peer_list		Mandatory. List of NPUs on the current AI server.
	local_id	Mandatory. Physical ID of an NPU. The value ranges from 0 to 64.
edge_count		Mandatory. Number of physical connection edges. The value ranges from 0 to UINT32_MAX.
edge_list		Mandatory. List of physical connection edges.
	net_layer	Mandatory. Network layer to which the current physical link belongs. The value ranges from 0 to 7.
	link_type	Mandatory. Connection type of the current physical link. The options are as follows: PEER2PEER PEER2NET
	protocols	Mandatory. List of protocols supported by the current link. The options are as follows: UB_CTP UB_TP ROCE HCCS TCP UB_MEM
	local_a	Mandatory. Physical ID of the NPU at one end of the communication link. The ID must exist in peer_list.
	local_a_ports	Mandatory. List of ports used by the communication link at the current layer of the NPU local_a. The value is a string of 1 to 32 characters, in the format of die ID/port ID. For example, "local_a_ports": ["1/0"] indicates port 0 on die 1.
	local_b	Mandatory. Physical ID of the NPU at the other end of the communication link. The ID must exist in peer_list.
	local_b_ports	Mandatory. List of ports used by the communication link at the current layer of the NPU local_b. The value is a string of 1 to 32 characters, in the format of die ID/port ID. Multiple ports are separated by commas (,). For example, "local_a_ports": ["1/0"] indicates port 0 on die 1.
	topo_instance_id	Optional. Topology instance ID. The value ranges from 0 to UINT32_MAX.
	topo_type	Optional. Topology type of the topology instance. The options are as follows: CLOS (default): All ranks can be connected, for example, in a fat-tree structure connected by switches. 1DMESH: Devices are directly connected to each other, forming a full-mesh topology.
	position	Optional. Location of the NIC used by the communication link. The options are as follows: HOST DEVICE (default)

Rank Table File Configuration

The following configuration uses two AI servers, each with two NPUs, as an example.

{
    "status": "completed",         // Rank table availability flag. The value completed indicates that the rank table is available.
    "version": "2.0",
    "rank_count": 4,
    "rank_list": [
        {
            "rank_id": 0,
            "local_id": 0,
            "device_id": 0,
            "device_port": 16666,
            "level_list":  [
                {
                    "net_layer": 0, // Intra-server connection
                    "net_instance_id": "az0-rack0-pod0",
                    "net_type": "TOPO_FILE_DESC",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.0.10",
                            "ports": ["1/0"] // Connection port (port 0 on die 1)
                        }
                    ]
                },{
                    "net_layer": 1, // Inter-server connection
                    "net_instance_id": "az0",
                    "net_type": "CLOS",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.0.15",
                            "ports": ["0/4","0/5","0/6","0/7"], // Connection port (ports 4, 5, 6, and 7 on die 0)
                            "plane_id": "plane0"
                        },{
                            "addr_type": "IPV4",
                            "addr": "172.16.0.5",
                            "ports": ["1/5","1/6"], // Connection port (ports 5 and 6 on die 1)
                            "plane_id": "plane1"
                        }
                    ]
                }
            ]
        },
        {
            "rank_id": 1,
            "local_id": 1,
            "device_id": 1,
            "device_port": 16667,
            "level_list": [
                {
                    "net_layer": 0,
                    "net_instance_id" : "az0-rack0-pod0",
                    "net_type": "TOPO_FILE_DESC",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.0.28",
                            "ports": ["1/0"]
                        }
                    ]
                },{
                    "net_layer": 1,
                    "net_instance_id": "az0",
                    "net_type": "CLOS",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.0.33",
                            "ports": ["0/4","0/5","0/6","0/7"],
                            "plane_id": "plane2"
                        },{
                            "addr_type": "IPV4",
                            "addr": "172.16.0.23",
                            "ports": ["1/5","1/6"],
                            "plane_id": "plane3"
                        }
                    ]
                }
            ]
        },{
            "rank_id": 2,
            "local_id": 0,
            "device_id": 0,
            "device_port": 16668,
            "level_list": [
                {
                    "net_layer": 0,
                    "net_instance_id": "az0-rack0-pod1",
                    "net_type": "TOPO_FILE_DESC",
                    "net_attr": "",
                    "rank_addr_list": [
                    {
                    "addr_type": "IPV4",
                    "addr": "172.16.1.10",
                    "ports": ["1/0"]
                    }
                    ]
                },{
                    "net_layer": 1,
                    "net_instance_id": "az0",
                    "net_type": "CLOS",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.1.15",
                            "ports": ["0/4","0/5","0/6","0/7"],
                            "plane_id": "plane0"
                        },{
                            "addr_type": "IPV4",
                            "addr": "172.16.1.5",
                            "ports": ["1/5","1/6"],
                            "plane_id": "plane1"
                        }
                    ]
                }
            ]
        },
        {
            "rank_id": 3,
            "local_id": 1,
            "device_id": 1,
            "device_port": 16669,
            "level_list": [
                {
                    "net_layer": 0,
                    "net_instance_id": "az0-rack0-pod1",
                    "net_type": "TOPO_FILE_DESC",
                    "net_attr": "",
                    "rank_addr_list": [
                    {
                    "addr_type": "IPV4",
                    "addr": "172.16.1.28",
                    "ports": ["1/0"]
                    }
                    ]
                },{
                    "net_layer": 1,
                    "net_instance_id": "az0",
                    "net_type": "CLOS",
                    "net_attr": "",
                    "rank_addr_list": [
                        {
                            "addr_type": "IPV4",
                            "addr": "172.16.1.33",
                            "ports": ["0/4","0/5","0/6","0/7"],
                            "plane_id": "plane2"
                        },{
                            "addr_type": "IPV4",
                            "addr": "172.16.1.23",
                            "ports": ["1/5","1/6"],
                            "plane_id": "plane3"
                        }
                    ]
                }
            ]
        }
    ]
}

**Table 2** Configuration items in the rank table file
Level-1 Configuration Item	Level-2 Configuration Item	Level-3 Configuration Item	Level-4 Configuration Item	Description
status				(Mandatory) Rank table availability flag. completed: The rank table is available. initializing: The rank table is unavailable.
version				Mandatory. Version of a rank table template. The value is fixed at 2.0.
rank_count				Mandatory. Number of ranks (or NPUs) involved in collective communication. The value ranges from 1 to 65536.
rank_list				Mandatory. List of ranks involved in collective communication.
	rank_id			Mandatory. Unique ID of a rank. The value ranges from 0 to number of NPUs – 1. To facilitate management, you are advised to sort the rank IDs based on the physical connection sequence of NPUs. That is, NPUs that are physically closer to each other are arranged together.
	local_id			Mandatory. Unique ID of an NPU in the current AI server. The value ranges from 0 to 64.
	device_id			Mandatory. Physical ID of an NPU. The value ranges from 0 to 64.
	device_port			Optional. Communication port of the device NIC. The value ranges from 1 to 65535. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. Set this field for a service with one device and multiple processes. Assign each process a unique port number to avoid conflicts.
	level_list			Mandatory. Resource information of a rank at each network layer. The length of the array in this list cannot exceed 8.
		net_layer		Mandatory. Network layer. The value ranges from 0 to 7. Number net_layer from 0, in ascending order.
		net_instance_id		Mandatory. Unique instance ID at the network layer. The value is user-defined, consisting of 1024 characters.
		net_type		Mandatory. Network type at the network layer. When net_layer is 0, this field can only be set to TOPO_FILE_DESC, indicating that the topology file is used for description. When net_layer is not 0, this field can be set to either of the following values: CLOS: All ranks can communicate with each other, for example, in a fat-tree structure connected by switches. TOPO_FILE_DESC: The network type is described by the topology file.
		net_attr		Optional. Reserved field, presenting additional information about the network layer.
		rank_addr_list		Mandatory. Network address used by the current rank at the network layer. The length of the array in this list cannot exceed 24. Each die must be configured separately.
			addr_type	Mandatory. Address type of the current rank. The options are as follows: EID IPv4 IPv6
			addr	Mandatory. IP address of the current rank. The value is a string of 1 to 256 characters, complying with the address format specified by addr_type.
			ports	Mandatory. List of ports bound to addr. An address can be bound to multiple ports, in the format of Die ID/Port ID. Multiple ports are separated by commas (,). Note: A port can be mapped to only one address at the same network layer. A maximum of 16 ports can be configured, each with a maximum of 32 characters.
			plane_id	Optional. Network plane ID. The default value is 0.

Parent topic: Cluster Configuration