Rank Table Configuration ( Atlas A3 training product / Atlas A3 inference product )

For the Atlas A3 training product / Atlas A3 inference product , cluster training supports both SuperPoD and typical networking. Each NPU of the Atlas A3 training product / Atlas A3 inference product contains two devices (two dies), and each device is a rank.

The rank table file is in JSON format. The comments in the JSON file provided in this section are for reference only. Delete the comments when you use it.

SuperPoD Networking

Assume that there are two SuperPoDs: each SuperPoD contains two AI servers, and each AI server contains four devices. The following is a file example describing the configuration.

{
    "status": "completed",         // Rank table availability flag. The value completed indicates that the rank table is available.
    "version": "1.2",              // Rank table template version. For the SuperPoD networking, set the version to 1.2.
    "server_count":"4",            // Number of AI servers participating in training.
    "server_list": [
        {
            "server_id": "node_0",     // AI server ID, which is of the string type. Ensure that the ID is globally unique.
            "host_ip":"172.16.0.100",  // Host IP address of the AI server
            "device": [
                {"device_id": "0","super_device_id":"0","device_ip": "192.168.1.6","device_port":"16666","backup_device_ip":"192.168.1.7","backup_device_port":"16667","host_port":"16665","rank_id": "0"}, // device_id: physical ID of the processor. super_device_id: physical ID of the processor in the SuperPoD system. device_ip: IP address of the processor's NIC. device_port: communication port of the processor's NIC. backup_device_ip: backup IP address when the operator retry feature between SuperPoDs is enabled. host_port: communication port of the host NIC. rank_id: rank ID, which starts from 0.
                {"device_id": "1","super_device_id":"1","device_ip": "192.168.1.7","device_port":"16666","backup_device_ip":"192.168.1.6","backup_device_port":"16667","host_port":"16666","rank_id": "1"},
                {"device_id": "2","super_device_id":"2","device_ip": "192.168.1.8","device_port":"16668","backup_device_ip":"192.168.1.9","backup_device_port":"16670","host_port":"16667","rank_id": "2"},
                {"device_id": "3","super_device_id":"3","device_ip": "192.168.1.9","device_port":"16669","backup_device_ip":"192.168.1.8","backup_device_port":"16667","host_port":"16668","rank_id": "3"}]
        },
        {
            "server_id": "node_1",
            "host_ip":"172.16.0.101",
            "device": [
                {"device_id": "0","super_device_id":"4","device_ip": "192.168.2.6","device_port":"16666","backup_device_ip":"192.168.2.7","backup_device_port":"16667","host_port":"16665","rank_id": "4"},
                {"device_id": "1","super_device_id":"5","device_ip": "192.168.2.7","device_port":"16666","backup_device_ip":"192.168.2.6","backup_device_port":"16667","host_port":"16666","rank_id": "5"},
                {"device_id": "2","super_device_id":"6","device_ip": "192.168.2.8","device_port":"16668","backup_device_ip":"192.168.2.9","backup_device_port":"16670","host_port":"16667","rank_id": "6"},
                {"device_id": "3","super_device_id":"7","device_ip": "192.168.2.9","device_port":"16669","backup_device_ip":"192.168.2.8","backup_device_port":"16667","host_port":"16668","rank_id": "7"}]
        },
        {
            "server_id": "node_2",
            "host_ip":"172.16.0.102",
            "device": [
                {"device_id":"0","super_device_id":"0","device_ip":"192.168.3.6","device_port":"16666","backup_device_ip":"192.168.3.7","backup_device_port":"16667","host_port":"16665","rank_id":"8"},
                {"device_id":"1","super_device_id":"1","device_ip":"192.168.3.7","device_port":"16666","backup_device_ip":"192.168.3.6","backup_device_port":"16667","host_port":"16666","rank_id":"9"},
                {"device_id":"2","super_device_id":"2","device_ip":"192.168.3.8","device_port":"16668","backup_device_ip":"192.168.3.9","backup_device_port":"16670","host_port":"16667","rank_id":"10"},
                {"device_id":"3","super_device_id":"3","device_ip":"192.168.3.9","device_port":"16669","backup_device_ip":"192.168.3.8","backup_device_port":"16667","host_port":"16668","rank_id":"11"}]
        },
        {
            "server_id": "node_3",
            "host_ip":"172.16.0.103",
            "device": [
                {"device_id":"0","super_device_id":"4","device_ip":"192.168.4.6","device_port":"16666","backup_device_ip":"192.168.4.7","backup_device_port":"16667","host_port":"16665","rank_id":"12"},
                {"device_id":"1","super_device_id":"5","device_ip":"192.168.4.7","device_port":"16666","backup_device_ip":"192.168.4.6","backup_device_port":"16667","host_port":"16666","rank_id":"13"},
                {"device_id":"2","super_device_id":"6","device_ip":"192.168.4.8","device_port":"16668","backup_device_ip":"192.168.4.9","backup_device_port":"16670","host_port":"16667","rank_id":"14"},
                {"device_id":"3","super_device_id":"7","device_ip":"192.168.4.9","device_port":"16669","backup_device_ip":"192.168.4.8","backup_device_port":"16667","host_port":"16668","rank_id":"15"}]
        }
    ],
    "super_pod_list": [
        {
            "super_pod_id": "0",          // Unique ID of the SuperPoD
            "server_list": [              // AI server list in the SuperPoD.
                {"server_id": "node_0"},  // server_id: server ID, which corresponds to server_id in server_list.
                {"server_id": "node_1"}]
        },
        {
            "super_pod_id": "1",
            "server_list": [
                {"server_id":"node_2"},
                {"server_id":"node_3"}]
        }
    ]
}

**Table 1** Rank table file description
Level-1 Configuration Item	Level-2 Configuration Item	Level-3 Configuration Item	Description
status			Mandatory. Rank table availability flag. completed: The rank table is available. initializing: The rank table is unavailable.
version			Mandatory. Version of a rank table template. For the SuperPoD networking, set this item to 1.2.
server_count			Optional. Number of AI servers that participate in collective communication.
server_list			Mandatory. List of AI servers that participate in collective communication.
	server_id		Mandatory. AI server ID. The value is a string with a maximum of 64 characters. Ensure that the value is globally unique. Example: node_0
	host_ip		Optional. Host IP address of the AI server. The value must be in IPv4 format. When the HCCL retry feature is enabled, this field must be configured. Otherwise, the retry feature does not take effect and the process without retry will be used. The retry feature is disabled by default. For details, see the environment variable HCCL_OP_RETRY_ENABLE.
	device		Mandatory. Device list.
		device_id	Mandatory. Physical ID of the AI processor, that is, the serial number of the device on the AI server. You can run the ls /dev/davinci* command to obtain the physical ID of the AI processor. For example, /dev/davinci0 indicates that the physical ID of the AI processor is 0. The value ranges from 0 to Actual number of devices – 1. Note: device_id takes precedence over the environment variable ASCEND_DEVICE_ID.
		super_device_id	Optional. If this field is not set, the AI server mode is used. Physical ID of the AI processor, which is the unique NPU ID in the SuperPoD system. You can run the npu-smi command to query the information. The following is an example: npu-smi info -t spod-info -i id -c *chip_id* id: device ID, which is the NPU ID obtained by running the npu-smi info -l command. chip_id: chip ID, which is obtained by running the npu-smi info -m command. In the command output, SDID is the unique NPU ID in the SuperPoD system.
		device_ip	Optional. IP address of the integrated NIC on the AI processor, which is globally unique and must be in the IPv4 or IPv6 format. Note: If the network contains multiple SuperPoDs, device_ip must be configured. If the network contains only one SuperPoD, device_ip must be configured in the following scenarios. In other scenarios, device_ip is optional. When RDMA is used for communication within a SuperPoD (that is, the environment variable HCCL_INTER_HCCS_DISABLE is set to TRUE to disable the HCCS function), device_ip must be configured. You can run the cat /etc/hccn.conf command on the current AI server to obtain the NIC IP address. address_0=xx.xx.xx.xx netmask_0=xx.xx.xx.xx netdetect_0=xx.xx.xx.xx address_xx shows the NIC IP address, in which xx is the physical ID of AI processor (device_id). The IP address following address_xx is the one you need to set for the device.
		device_port	Optional. Communication port of the device NIC. The value ranges from 1 to 65535. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. Set this field for a service with one device and multiple processes. Assign each process a unique port number to avoid conflicts.
		backup_device_ip	Optional. When the operator retry feature is enabled for communication between SuperPoDs, you can set another die NIC on the same NPU for a backup if the device NIC fails (RDMA link fault). This improves the operator retry success rate. This communication mode is called link failover communication. The value of backup_device_ip is in IPv4 or IPv6 format. For details about how to query the value, see the configuration description of device_ip. Note: The devices corresponding to backup_device_ip and device_ip must belong to the same NPU. That is, only the device NICs corresponding to die 0 and die 1 on the same NPU can serve as the backup for each other. This configuration takes effect only when the communication operator expansion mode is AI_CPU and the operator retry feature is enabled for communication between SuperPoDs. That is: export HCCL_OP_EXPANSION_MODE="AI_CPU" export HCCL_OP_RETRY_ENABLE="L1:1,L2:1" L2 indicates an inter-SuperPoD communicator, and the value 1 indicates that the operator retry feature is enabled. To ensure that the link failover communication can be properly executed, the following conditions must be met: The communication link of the standby NIC is normal. Both devices in active/standby mode must be visible to services. For example, NPU 1 contains two dies, Device 0 and Device 1, which work in active/standby mode. If only device 0 is visible to services (specified through ASCEND_RT_VISIBLE_DEVICES) while device1 is not, the link failover communication cannot work. For details about the environment variable ASCEND_RT_VISIBLE_DEVICES, see Environment Variables. If link failover occurs in a communication process (assuming that the die 0 NIC of an NPU is faulty, and the standby die 1 NIC is enabled), traffic of the original die 0 NIC is sent and received using the die 1 NIC. As a result, traffic of die 1 increases, and the overall performance deteriorates due to halved physical bandwidth and port conflicts. In the scenario where link failover communication is enabled, if the die 0 NIC of NPU 0 is faulty, the standby die 1 NIC is used. Because the communication between two NPUs requires that both the local and remote ends be switched to the standby NICs, NPU 1 is also switched from die 0 to die 1, as shown in Figure 1. However, if there is a communication task between die 0 and die 1, link failover cannot be conducted. When the link failover communication is enabled, it is recommended that two dies of one NPU be allocated to the same training or inference task. If two dies of one NPU are allocated to two different training or inference tasks and one task is faulty, the NIC of the other task is used. As a result, the performance of both tasks deteriorates to some extent. One NPU supports only one link failover does not support failback. As shown in Figure 2, the communication link between NPU 0 and NPU 1 is faulty in Case 1, and the standby link is used. Link failover is enabled, so the communication is normal. If the fault recurs in Case 2, link failover is not supported. The system reports an error and exits.
		backup_device_port	Optional. Communication port of the standby device NIC. The value ranges from 1 to 65535. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. Set this field for a service with one device and multiple processes in the link failover communication. Assign each process a unique port number to avoid conflicts. Note: The communication port numbers must differ when the same device NIC serves as both the active and standby NICs.
		host_port	Optional. Communication port of the host NIC. The value ranges from 1 to 65535. The host_port value must be different for devices in the same AI server. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. If the HCCL retry feature is enabled by setting the environment variable HCCL_OP_RETRY_ENABLE and the service runs in the single-device multi-process scenario (that is, multiple service processes share one NPU), you are advised to set this field. Assign each process a unique port number to avoid conflicts.
		rank_id	Mandatory. Unique ID of a rank. The value ranges from 0 to number of devices – 1. You are advised to sort rank_id based on the physical connection sequence of devices, that is, arrange devices by physical connection proximity. Otherwise, the performance may be affected. For example, if device_ip is set in ascending order based on physical connections, you are advised to set rank_id in ascending order. rank_id must be sequentially allocated within each AI server and cannot be interleaved across servers. Correct: The rank_id set is {0, 1, 2, 3} for server 1, and {4, 5, 6, 7} for server 2. Incorrect: The rank_id set is {0, 1, 2, 7} for server 1, and {3, 4, 5, 6} for server 2.
super_pod_list			Optional. If this field is not set, the AI server mode is used. List of SuperPoDs that participate in collective communication.
	super_pod_id		This field is mandatory if super_pod_list is configured. Unique ID of a SuperPoD, which is globally unique. The following configuration modes are supported: Set this field to the physical ID of the SuperPoD. You can use the npu-smi tool to query the physical ID. The following is a command example: npu-smi info -t spod-info -i id -c *chip_id* id: device ID, which is the NPU ID obtained by running the npu-smi info -l command. chip_id: chip ID, which is obtained by running the npu-smi info -m command. In the command output, Super Pod ID indicates the physical ID of the SuperPoD. Define a custom ID, which must be a globally unique string. In this scenario, you can split a physical SuperPoD into multiple small logical SuperPoDs. For example, if a physical SuperPoD has eight AI servers, you can use the first four AI servers as a small SuperPoD named super_pod_1, and the last four AI servers as another small SuperPoD named super_pod_2.
	server_list		Mandatory. List of AI servers on a SuperPoD.
		server_id	Mandatory. Server ID. The value is a string corresponding to server_id in server_list. Example: node_0

If there are multiple SuperPoDs on the network, configure information about the AI servers that belong to the same SuperPoD together. Assume that there are two SuperPoDs whose IDs are 0 and 1. Configure AI servers on SuperPoD 0, and then configure AI servers on SuperPoD 1. Cross-configuring AI servers on SuperPoDs 0 and 1 is not supported.

Figure 1 Link failover communication

Figure 2 One NPU allows only one link failover

Typical Cluster Networking (AI Server Mode)

The following is an example of configuring the rank table file when there are two AI servers and each AI server has two devices:

{
    "status":"completed",   // Rank table availability flag. The value completed indicates that the rank table is available.
    "version":"1.0",        // Rank table template version. For typical cluster networking, set this option to 1.0.
    "server_count":"2",     // Number of AI servers participating in training. In this example, there are two AI servers.
    "server_list":
    [
        {
            "server_id":"node_0",       // AI server ID, which is of the string type. Ensure that the ID is globally unique.
            "host_ip":"172.16.0.110",   // Host IP address of the AI server
            "device":[   // List of devices on the AI server
                {
                    "device_id":"0",              // Physical ID of the processor
                    "device_ip":"192.168.1.8",    // NIC IP address of the processor
                    "device_port":"16667",        // Communication port of the processor's NIC
                    "host_port":"16666",          // Communication port of the host NIC
                    "rank_id":"0"                 // Rank ID, starting from 0
                },
                {
                    "device_id":"1",
                    "device_ip":"192.168.1.9", 
                    "device_port":"16667",
                    "host_port":"16667", 
                    "rank_id":"1"
                }
            ]
        },
        {
            "server_id":"node_1",
            "host_ip":"172.16.0.111",
            "device":[
                {
                    "device_id":"0",
                    "device_ip":"192.168.2.8",
                    "device_port":"16667",
                    "host_port":"16666", 
                    "rank_id":"2"
                },
                {
                    "device_id":"1",
                    "device_ip":"192.168.2.9", 
                    "device_port":"16667",
                    "host_port":"16667", 
                    "rank_id":"3"
                }
            ]
        }
    ]
}

**Table 2** Rank table file description
Level-1 Configuration Item	Level-2 Configuration Item	Level-3 Configuration Item	Description
status			Mandatory. Rank table availability flag. completed: The rank table is available. initializing: The rank table is unavailable.
version			Mandatory. Version of a rank table template. For the typical cluster networking, set this item to 1.0.
server_count			Mandatory. Number of AI servers that participate in collective communication.
server_list			Mandatory. List of AI servers that participate in collective communication.
	server_id		Mandatory. AI server ID. The value is a string with a maximum of 64 characters. Ensure that the value is globally unique. Example: node_0
	host_ip		Optional. Host IP address of the AI server. The value must be in IPv4 format. When the HCCL retry feature is enabled, this field must be configured. Otherwise, the retry feature does not take effect and the process without retry will be used. The retry feature is disabled by default. For details, see the environment variable HCCL_OP_RETRY_ENABLE.
	device		Mandatory. List of devices on the AI server.
		device_id	Mandatory. Physical ID of the AI processor, that is, the serial number of the device on the AI server. You can run the ls /dev/davinci* command to obtain the physical ID of the AI processor. For example, /dev/davinci0 indicates that the physical ID of the AI processor is 0. The value ranges from 0 to Actual number of devices – 1. Note: device_id takes precedence over the environment variable ASCEND_DEVICE_ID.
		device_ip	Optional. IP address of the integrated NIC on the AI processor, which is globally unique and must be in the IPv4 or IPv6 format. Note: In the multi-server scenario, device_ip is required. In the single-server scenario, device_ip is optional. You can run the cat /etc/hccn.conf command on the current AI server to obtain the IP address of the NIC. address_0=xx.xx.xx.xx netmask_0=xx.xx.xx.xx netdetect_0=xx.xx.xx.xx address_xx shows the NIC IP address, in which xx is the physical ID of AI processor (device_id). The IP address following address_xx is the one you need to set for the device.
		device_port	Optional. Communication port of the device NIC. The value ranges from 1 to 65535. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. Set this field for a service with one device and multiple processes. Assign each process a unique port number to avoid conflicts.
		host_port	Optional. Communication port of the host NIC. The value ranges from 1 to 65535. The host_port value must be different for devices in the same AI server. Ensure that the port is not occupied by other processes. Ports 1–1023 are reserved. Do not use these ports. If the HCCL retry feature is enabled by setting the environment variable HCCL_OP_RETRY_ENABLE and the service runs in the single-device multi-process scenario (that is, multiple service processes share one NPU), you are advised to set this field. Assign each process a unique port number to avoid conflicts.
		rank_id	Mandatory. Unique ID of a rank. The value ranges from 0 to number of devices – 1. You are advised to sort rank_id based on the physical connection sequence of devices, that is, arrange devices by physical connection proximity. Otherwise, the performance may be affected. For example, if device_ip is set in ascending order based on physical connections, you are advised to set rank_id in ascending order. rank_id must be sequentially allocated within each AI server and cannot be interleaved across servers. Correct: The rank_id set is {0, 1, 2, 3} for server 1, and {4, 5, 6, 7} for server 2. Incorrect: The rank_id set is {0, 1, 2, 7} for server 1, and {3, 4, 5, 6} for server 2.

Parent topic: Cluster Configuration