Introduction

Application Scenario

To test the functional correctness and performance of Huawei Collective Communication Library (HCCL) collective communication in distributed training scenarios, you can use the HCCL Performance Tester.

This tool supports only the network performance test of collective communication implemented based on the HCCL single-operator API.

Environment Setup

The compilation and running of the HCCL Performance Tester depend on the CANN development kit (Ascend-CANN-Toolkit) and operator package (Ascend-cann-ops). Therefore, before using this tool, ensure that relevant CANN software packages have been deployed in the environment. For details about the installation, see CANN Software Installation.
The source code of the HCCL Performance Tester is stored in ${INSTALL_DIR}/tools/hccl_test. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.

Supported Products

Atlas 350 Accelerator Card

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product

Atlas training product

Atlas inference product

For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server, Atlas 300I A2 inference card, and A200I A2 Box heterogeneous components are supported.

For Atlas inference product, only the Atlas 300I Duo Inference Card is supported.

Restrictions

Atlas 350 Accelerator Card: HCCL Performance Tester supports a maximum cluster networking scenario containing 32,000 communication ranks.
Atlas A3 training product/Atlas A3 inference product: HCCL Performance Tester supports a maximum cluster networking scenario containing 32,000 communication ranks.
AlltoAll and AlltoAllV operators: HCCL Performance Tester supports a maximum cluster networking scenario containing 8,000 communication ranks.
Atlas A2 training product/Atlas A2 inference product: HCCL Performance Tester supports a maximum cluster networking scenario containing 32,000 communication ranks.
Atlas training product: HCCL Performance Tester supports a maximum cluster networking scenario containing 4,096 communication ranks.

Background Knowledge

The collective communication bandwidth refers to the "algorithm bandwidth", that is, the data volume/time consumed when a specific collective communication operation is performed.

For example, if the AllReduce operation is performed on eight devices on a single server, the data volume divided by the time consumed to complete the AllReduce operation is the algorithm bandwidth of the AllReduce operator execution.

When the HCCL performance tester is used for testing, the bandwidth data refers to the algorithm bandwidth.

The main factors affecting the algorithm bandwidth include the physical bandwidth of the communication link and the orchestration and implementation of the communication algorithm itself.