HcclCommWorkingDevNicSet

Applicability

For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800T A2 training server, Atlas 900 A2 PoD cluster basic unit, and Atlas 200T A2 Box16 heterogeneous subrack are supported.

Function

Configures the communication NIC in the communicator in the cluster scenario. Communication can be switched between a device NIC and the standby NIC on the same NPU. The standby NIC is another die NIC on the same NPU.

When configuring the NIC for a single communication, this API needs to be called for all devices in the same communicator, and the ranks, useBackup, and nRanks parameters delivered by all devices must be the same.

When configuring the communication NIC, pay attention to the following:

• When the commands for switching to the standby NIC and active NIC are dispatched at the same time, the commands fail to be executed if one end of the link is switched to the active NIC and the other end is switched to the standby NIC.
• If there is a link between device 1 and device 2, the first command is dispatched to switch device 1 to the standby NIC, and the second command is dispatched to switch device 2 to the default NIC. In this case, the link between device 1 and device 2 uses the default NIC for communication, and the link between device 1 and other devices uses the standby NIC for communication.
• If the NICs of device 1 and device 2 are standby NICs of each other and both devices dispatch the command for switching to the standby NIC, the communication NICs used by device 1 and device 2 are exchanged.

Prototype

1

HcclResult HcclCommWorkingDevNicSet(HcclComm comm, uint32_t *ranks, bool *useBackup, uint32_t nRanks)

Parameters

Returns

HcclResult: HCCL_SUCCESS on success, or else failure.

Constraints

• Before the configuration, ensure that the communication task in the current environment is complete.
• When ranks in the same communicator call this API, the lengths of the passed ranks and useBackup arrays must be the same as the value of nRanks.
• This API triggers an actual NIC switchover only when the following conditions are met:
  • The retry feature is enabled for communication between SuperPoDs. That is, L2 of the environment variable HCCL_OP_RETRY_ENABLE is set to 1.
  • The RDMA link exists in the topology.
• In unsupported scenarios, for example, when retry is not enabled using HCCL_OP_RETRY_ENABLE, HCCL_SUCCESS is returned and WARNING is recorded in the log. However, the NIC is not actually switched.
• For the same rank, HcclCommWorkingDevNicSet APIs need to be called one by one in sequence. Concurrent call is not allowed.
• For the entire communicator, ensure that HcclCommWorkingDevNicSet APIs are called in the same sequence among different ranks.
• The NIC can be configured only when the communicator to which the comm handle points has dispatched operators. The same NIC configuration is used for the operators of the same type and with the same parameters that are dispatched later. Other newly dispatched operators use the default NIC for communication. If the preceding requirements are not met, the NIC configuration fails.

Call Example

1
2
3
4
5

HcclComm comm;
uint32_t rankSize = 4;
uint32_t rankIds[rankSize] = {0, 3, 7, 12};
bool useBackup[rankSize] = {true, true, true, true};
HcclCommWorkingDevNicSet(comm, rankIds, useBackup, rankSize);