HCCL_OP_EXPANSION_MODE

Description

Configures the expansion mode of communication operators.

The following lists supported configurations and corresponding scenario descriptions for each product model. Products not listed do not support this environment variable. If an unsupported environment variable is set, the default value is used.

• For the Atlas 350 Accelerator Card: By default, the adaptive expansion mode is used. A proper mode is automatically selected based on the networking topology, data volume, and other factors. The available options of this environment variable are as follows:
  • AI_CPU: Communication operators are expanded on the AI CPU. The device side automatically selects the corresponding scheduler based on the hardware model.

    This configuration only supports the following operators: Broadcast, Reduce, AllReduce, Scatter, ReduceScatter, AllGather, AlltoAll, AlltoAllV, and AlltoAllVC.

    Notes:

    In graph mode (Ascend IR) or graph capture (aclgraph) scenarios, if the communication algorithm uses the AI_CPU mode, the number of concurrent graphs on a single device cannot exceed 6. Otherwise, communication congestion may occur due to full occupation of AI CPU cores.

  • AIV: Communication operators are expanded and executed on Vector Cores. This configuration is not supported in the current version.
    • This configuration option supports only the symmetric networking and inference features.
    • This configuration option does not support the multi-communicator parallel scenario (because the AIV mode cannot be used for multiple communicators at the same time). Otherwise, unexpected behavior may occur. When initializing a communicator with specific configurations, you can set the operator expansion mode of a specific communicator to AIV through HcclCommConfig.
    • This configuration only supports Broadcast, Reduce, AllReduce, ReduceScatter, Scatter, AllGather, AlltoAll, and AlltoAllV operators, and is limited to single-server scenarios only.
      • For the Broadcast, Scatter, AllGather, AlltoAll, and AlltoAllV operators, the supported data types include int8, uint8, int16, uint16, int32, uint32, int64, uint64, float16, float32, and bfp16.
      • For the Reduce, AllReduce, and ReduceScatter operators, the supported data types include int8, int16, int32, float16, float32, and bfp16.
    • Under this configuration, AllReduce, ReduceScatter, AllGather, and AlltoAll operators support core control capabilities. You are advised to configure the number of Vector Cores according to the concurrency of compute and communication operators in actual service scenarios.

      If the number of Vector Cores allocated during service compilation fails to meet the algorithm orchestration requirements, HCCL will throw an error and prompt the minimum number of Vector Cores required.

  • CCU_MS: Communication operators are expanded on the CCU in Memory Slice (MS) mode. This configuration is not supported in the current version.

    In MS mode, the on-chip CCU MS serves as a transfer buffer for communication with multiple remote ends to reduce memory read/write bandwidth consumption. The MS features small capacity but high access speed.

    The system will automatically fall back to the AI_CPU mode when CCU resources are insufficient.

  • CCU_SCHED: Communication operators are expanded on the CCU in scheduling mode.

    In this mode, the CCU acts as a scheduler to schedule UB WQE tasks to the UB engine. The on-chip CCU MS is not used; data is transmitted directly between HBM of two ranks.

    For the AllReduce, ReduceScatter, and Reduce operators in single-server communication, once the data volume exceeds a certain threshold, the system will automatically switch to the AI_CPU mode to avoid performance degradation. This threshold is not fixed and varies with factors such as the operator execution mode and network scale.

    The system will automatically fall back to the AI_CPU mode when CCU resources are insufficient.

• For the Atlas A3 training product / Atlas A3 inference product , the following options are supported:
  • AI_CPU (default): Communication operators are expanded on the AI CPU. The device side automatically selects the corresponding scheduler based on the hardware model.

    Full communication operators are supported within a supernode and between supernodes. For the Reduce, ReduceScatter, ReduceScatterV, and AllReduce operators, the data type can only be int8, int16, int32, float16, float32, or bfp16, and the reduce operation type can only be sum, max, or min. For details about the data types supported by other communication operators, see the corresponding collective communication APIs.

    Notes:

    • In the graph mode (Ascend IR) or graph capture (aclgraph) scenario, when the communication algorithm uses the default AI CPU mode, the number of concurrent graphs on a single device cannot exceed 6. Otherwise, the communication may be blocked because the AI CPU cores are fully occupied.
    • In this mode, the communication function depends on the open AI CPU user mode to deliver scheduling tasks, which poses security risks. You need to ensure the security and reliability of custom operators to prevent malicious attacks.
  • AICPU_CacheDisable: Disables the AI CPU cache feature for HCCL operators.

    AI CPU cache allows HCCL to reuse the execution results generated during the first run of a communication operator when the same operator is executed again, reducing operator expansion overhead. Enabling this feature incurs additional device memory overhead. Therefore, in service scenarios with frequently changing communication data sizes, disable this feature to reduce device memory consumption.

  • AIV: Communication operators are expanded and executed on Vector Cores.
    • This configuration option supports only the symmetric networking and inference features.
    • This configuration option does not support the multi-communicator parallel scenario (because the AIV mode cannot be used for multiple communicators at the same time). Otherwise, unexpected behavior may occur. When initializing a communicator with specific configurations, you can set the operator expansion mode of a specific communicator to AIV through HcclCommConfig.
    • This configuration option supports only the Broadcast, AllReduce, ReduceScatter, AllGather, AlltoAll, AlltoAllV, and AlltoAllVC operators.
      • For the Broadcast operator, supported data types include int8, uint8, int16, uint16, int32, uint32, float16, float32, and bfp16. It only supports single-server communication within a supernode and single-operator mode. Multi-server and inter-supernode communication are not supported.
      • For the AllReduce operator, the data type can be int8, int16, int32, float16, float32, or bfp16. The reduce operation type can only be sum, max, or min. Only single-server or multi-server communication within a supernode is supported. Communication between supernodes is not supported.
      • For the ReduceScatter operator, the data type can be int8, int16, int32, float16, float32, or bfp16. The reduce operation type can only be sum, max, or min. Only single-server or multi-server communication within a supernode is supported. Communication between supernodes is not supported.
      • For the AllGather, AlltoAll, AlltoAllV, and AlltoAllVC operators, the data type can be int8, uint8, int16, uint16, int32, uint32, float16, float32, or bfp16. Only single-server or multi-server communication within a supernode is supported. Communication between supernodes is not supported.
    • For Broadcast, AllReduce, ReduceScatter, AllGather, and AlltoAll operators in single-server communication scenarios, the system automatically switches to the AI_CPU mode once the communication data volume exceeds a certain threshold to avoid performance degradation. The threshold is not fixed and varies with factors such as the operator execution mode, whether deterministic computation is enabled, and network scale. For AlltoAllV, AlltoAllVC, and AlltoAll operators in multi-server communication scenarios, if the AIV mode is configured, the system will not automatically switch to the AI_CPU mode. To prevent performance degradation, you are advised to use the AIV mode when the maximum communication data volume between any two ranks is no more than 1 MB; otherwise, use the AI_CPU mode.
    • Under this configuration option, the collective communication supports the core control capability. You are advised to configure the number of vector cores based on the concurrency of compute operators and communication operators in actual application scenarios.
      • For the Broadcast operator, you are advised to allocate at least ranksize vector cores.
      • For the AllReduce, ReduceScatter, AllGather, AlltoAll, AlltoAllV, and AlltoAllVC operators, you are advised to allocate at least max(2, ranksize/20 + 1) vector cores.

      If the number of Vector Cores allocated during service compilation fails to meet the algorithm orchestration requirements, HCCL will throw an error and prompt the minimum number of Vector Cores required.

    Notes:

    When the algorithm orchestration expansion location is set to AIV, if the HCCL_DETERMINISTIC environment variable is set to true or strict to enable deterministic computation, deterministic computation takes higher priority, and the AIV expansion may fail to take effect in certain scenarios.

• For the Atlas A2 training product / Atlas A2 inference product , the following options are supported:
  • HOST (default): Communication operators are expanded on the host CPU. The device side automatically selects the corresponding scheduler based on the hardware model.
  • HOST_TS: Communication operators are expanded on the host CPU. The host delivers tasks to the task scheduler on the device, and the scheduler schedules and executes the tasks.
  • AI_CPU: Communication operators are expanded on the AI CPU. The device side automatically selects the corresponding scheduler based on the hardware model.

    This option supports only the AllGather, AlltoAll, AlltoAllV, and AlltoAllVC operators.

    Notes:

    In graph mode (Ascend IR) or graph capture (aclgraph) scenarios, if the communication algorithm uses the AI_CPU mode, the number of concurrent graphs on a single device cannot exceed 6. Otherwise, communication congestion may occur due to full occupation of AI CPU cores.

  • AIV: Communication operators are expanded and executed on Vector Cores.
    • This configuration option supports only the symmetric networking and inference features.
    • This configuration option does not support the multi-communicator parallel scenario (because the AIV mode cannot be used for multiple communicators at the same time). Otherwise, unexpected behavior may occur. When initializing a communicator with specific configurations, you can set the operator expansion mode of a specific communicator to AIV through HcclCommConfig.
    • This configuration option supports only the Broadcast, AllReduce, AlltoAll, AlltoAllV, AlltoAllVC, AllGather, ReduceScatter, AllGatherV, and ReduceScatterV operators.
      • For the Broadcast operator, the data type can be int8, uint8, int16, uint16, int32, uint32, float16, float32, or bfp16. Only the single-operator mode with eight or fewer devices in the single-server scenario is supported.
      • For the AllReduce operator, the data type can be int8, int16, int32, float16, float32, or bfp16. The reduce operation type can only be sum, max, or min.
      • For the AlltoAll, AlltoAllV, and AlltoAllVC operators, the data type can be int8, uint8, int16, uint16, int32, uint32, float16, float32, or bfp16. For the AlltoAllV and AlltoAllVC operators, only single-server scenarios are supported. The graph mode of the AlltoAll operator supports only single-server scenarios.
      • For the AllGather operator, the data type can be int8, uint8, int16, uint16, int32, uint32, float16, float32, or bfp16. The graph mode of this operator supports only single-server scenarios.
      • For the ReduceScatter operator, the data type can be int8, int16, int32, float16, float32, or bfp16. The reduce operation type can only be sum, max, or min. The graph mode of this operator supports only single-server scenarios.
      • For the AllGatherV operator, the data type can be int8, uint8, int16, uint16, int32, uint32, float16, float32, or bfp16. Only the single-operator mode is supported.
      • For the ReduceScatterV operator, the data type can be int8, int16, int32, float16, float32, or bfp16. The reduce operation type can only be sum, max, or min.
    • Under this configuration option, the collective communication supports the core control capability. You are advised to configure the number of vector cores based on the concurrency of compute operators and communication operators in actual application scenarios.
      • For the AllReduce, ReduceScatter, and ReduceScatterV operators, you are advised to allocate at least 24 cores.
      • For the Broadcast, AlltoAll, AlltoAllV, AlltoAllVC, AllGather, and AllGatherV operators, you are advised to allocate at least 16 cores.

      If the number of Vector Cores allocated during service compilation fails to meet the algorithm orchestration requirements, HCCL will throw an error and prompt the minimum number of Vector Cores required.

    Notes:

    • When the algorithm orchestration expansion location is set to AIV, if the HCCL_DETERMINISTIC environment variable is set to true or strict to enable deterministic computation, deterministic computation takes higher priority, and the AIV expansion may fail to take effect in certain scenarios.
    • The Atlas 200T A2 Box16 heterogeneous subrack does not support inter-subrack communication.
• For the Atlas 300I Duo Inference Card, the following options are supported:
  • HOST (default): Communication operators are expanded on the host CPU. The device side automatically selects the corresponding scheduler based on the hardware model.
  • AI_CPU: Communication operators are expanded on the AI CPU. The device side automatically selects the corresponding scheduler based on the hardware model.
    • Only the single-server single-communicator scenario is supported.
    • Only the AllReduce operator is supported. For details about the data types supported by the AllReduce operator, see the HcclAllReduce API.
    • If this parameter is set to AI_CPU, the communication operator does not support profile data collection and analysis.
    • For static shape graphs, this option is not supported, meaning you cannot set the communication operator expansion mode to AI_CPU.

Example

export HCCL_OP_EXPANSION_MODE="HOST"

Constraints

• If you call the HCCL C APIs to initialize a communicator with specific configurations and set the communication operator expansion mode via the hcclOpExpansionMode parameter in HcclCommConfig, the configuration at the communicator granularity takes precedence.
• For the inference feature of the Atlas A2 training products / Atlas A2 inference products :
  If AIV is configured and the process is forcibly ended by pressing Ctrl+C, the device log file exported by the msnpureport tool may contain an error indicating that the device accesses an invalid address. The log keyword is devmm_page_fault_d2h_query_flag, devmm_svm_device_fault, or ipc_fault_msg_para_check, as shown in the following. This scenario does not affect the device status or the execution of new tasks.

  1 2 3 4 5

  [ERROR] KERNEL(5044,sklogd):2024-07-29-10:33:22.646.254 [klogd.c:247][257382.266115] [ascend] [ERROR] [devmm] [devmm_page_fault_d2h_query_flag 810] <kworker/u16:2:14887,14887> Host page fault send message fail.(hostpid=2131021; devid=0; vfid=0; ret=-22; va=0x12c700300000; hostpid=2131021; devid=0; vfid=0) [ERROR] KERNEL(5044,sklogd):2024-07-29-10:33:22.646.284 [klogd.c:247][257382.266124] [ascend] [ERROR] [devmm] [devmm_svm_device_fault 468] <kworker/u16:2:14887,14887> Vm fault failed. (hostpid=2131021; devid=0; vfid=0; ret=64; fault_addr=0x12c700300000; start=0x12c700300000) [ERROR] KERNEL(5044,sklogd):2024-07-29-10:33:22.659.429 [klogd.c:247][257382.282181] [ascend] [ERROR] [tsdrv] [ipc_fault_msg_para_check 309] <swapper/3:0> Invalid node id. (devid=0; node_type=100; node_id=40; node_num=25) ................ [ERROR] KERNEL(5044,sklogd):2024-07-29-10:33:24.874.211 [klogd.c:247][257384.473533] [ascend] [ERROR] [tsdrv] [tsdrv_hb_cq_callback 332] <kworker/0:0:20353> receive ts exception msg, call excep_code=0xb4060006, time=1722249204.850014098s, devid=0 tsid=0