System Constraints and Limitations

Common Constraints

Collective communication is not applicable to Ascend virtual instance scenarios.

An Ascend virtual instance can virtualize an NPU of a physical machine or virtual machine into multiple virtual NPUs (vNPU instances) and mount the vNPU instances to target environments. Virtualization management allocates and reclaims resources of different specifications in a unified manner, allowing you to allocate and deallocate resources repeatedly.

Atlas 350 Accelerator Card

For the Ascend computing module to adapt to third-party x86 servers, the OS kernel version must be later than 5.18. If this requirement is not met, you need to manually install the entropy supplementation tool haveged and start its service.

Atlas A3 Training Product / Atlas A3 Inference Product

• If the driver firmware version is 25.0.RC1 or later, the single-device multi-process service scenario is supported. That is, multiple service processes can share one NPU. However, managing multiple processes incurs significant resource overhead and compromises communication performance. When excessive processes are allocated to the same NPU, service failures may occur due to resource exhaustion. If your driver firmware does not meet the version requirements, a single process will be used.

  If the communication algorithm orchestration is expanded in AICPU mode (default value), it is recommended that the number of concurrent processes on a single device does not exceed 6. If the communication algorithm orchestration is expanded in AIV mode, it is recommended that multiple processes on a single device be executed in serial mode. Follow the preceding suggestions to avoid task deadlock. You can set the expansion mode of the communication algorithm using the environment variable HCCL_OP_EXPANSION_MODE.

• It is recommended that each SuperPoD have the same number of servers and each server have the same number of AI processors. Otherwise, the performance degrades.
• In graph (Ascend IR) or graph capture (ACLGraph) mode, when the communication algorithm uses the default AICPU mode, the number of concurrent graphs on a single device cannot exceed 6. Otherwise, the communication may be blocked because the AICPU cores are fully occupied.

Atlas A2 Training Product / Atlas A2 Inference Product

• If the driver firmware version is 25.0.RC1 or later, the single-device multi-process service scenario is supported. That is, multiple service processes can share one NPU. If the network contains MC² operators (operators that integrate computing and communication, such as AllGatherMatmul, MatmulReduceScatter, and AlltoAllAllGatherBatchMatMul), the single-device multi-process service scenario is not supported. However, managing multiple processes incurs significant resource overhead and compromises communication performance. When excessive processes are allocated to the same NPU, service failures may occur due to resource exhaustion. If your driver firmware does not meet the version requirements, a single process will be used.

  If the communication algorithm orchestration is expanded in HOST mode (default value), it is recommended that the number of concurrent processes on a single device does not exceed 8. If the communication algorithm orchestration is expanded in AIV mode, it is recommended that multiple processes on a single device be executed in serial mode. Follow the preceding suggestions to avoid task deadlock. You can set the expansion mode of the algorithm using the environment variable HCCL_OP_EXPANSION_MODE.

• In single-server scenarios, the number of AI processors that participate in collective communication is not limited. In server cluster scenarios, the number of AI processors that participate in collective communication must be (1 to 8) × n (n is the number of servers participating in training). It is recommended that each server should have the same number of AI processors that participate in collective communication. Otherwise, the performance deteriorates.

Atlas Training Product

• The single-device multi-process service scenario is not supported. That is, multiple service processes cannot share one NPU.
• In single-server scenarios, the number of AI processors that participate in collective communication can only be 1, 2, 4, or 8. In addition, devices 0 to 3 and devices 4 to 7 form separate networks. When two or four devices are used for training, cross-network clusters cannot be created. In server cluster scenarios, the number of AI processors that participate in collective communication can only be 1 × n, 2 × n, 4 × n, or 8 × n (n is the number of servers participating in training). If n is a power of 2, the cluster performance is the best. Therefore, this mode is recommended for cluster networking.

Atlas 300I Duo Inference Card

• The single-device multi-process service scenario is not supported. That is, multiple service processes cannot share one NPU.
• Only the single-server scenario is supported. For details about the maximum number of NPUs supported by each collective communication operation, see Collective Communication.
• The communicator must be initialized before any other operations that require device memory allocation. Otherwise, the initialization may fail due to insufficient P2P memory.