Precautions

Before performing distributed training, refer to this section for some precautions.

Before starting distributed training across multiple processes, you need to configure the resource information of AI processors that participate in the distributed training.

Currently, resource information can be configured using configuration files or environment variables. You can choose either of them, but they cannot be used together.

Using configuration files. The resource configuration file, known as a rank table file, is used alongside environment variables such as RANK_TABLE_FILE and RANK_ID.
For details about how to configure resource information and start the training process, see Training Execution (Configuring Resources via the Rank Table).
Setting environment variables
For details about how to configure resource information and start the training process, see Training Execution (Configuring Resources Via Environment Variables).

Before performing distributed training, pay attention to the following points:

Atlas training product: In single-server scenarios, the number of AI processors that participate in collective communication can be 1, 2, 4, or 8. In addition, devices 0 to 3 and devices 4 to 7 form separate networks. When two or four devices are used for training, cross-network clusters cannot be created. In server cluster scenarios, the number of AI processors that participate in collective communication can only be 1 x n, 2 x n, 4 x n, or 8 x n (n is the number of servers participating in training). If n is an exponential multiple of 2, the cluster performance is the best. Therefore, this mode is recommended for cluster networking.
Atlas A2 training product/Atlas A2 inference product: In single-server scenarios, the number of AI processors that participate in collective communication is not limited. In server cluster scenarios, the number of AI processors that participate in collective communication must be (1 to 8) x n (n is the number of servers participating in training). It is recommended that each server should have the same number of AI processors that participate in collective communication. Otherwise, the performance deteriorates.
Atlas A3 training product/Atlas A3 inference product: It is recommended that each supernode should have the same number of servers and each server should have the same number of AI processors. Otherwise, the performance deteriorates.
Each device corresponds to a training process. It is not supported to run multiple training processes on a single device.

Parent topic: Distributed Training with Multiple Devices