HCCL_DETERMINISTIC
Description
Enables or disables the deterministic computation or order-preserving function for reduction communication operators, including AllReduce, ReduceScatter, ReduceScatterV, and Reduce. Order-preserving reduction refers to strict deterministic computation, which guarantees uniform reduction order while maintaining deterministic computation.
When the deterministic computation or order-preserving function is enabled for a reduction operator, the same output is generated if the operator is executed for multiple times with the same hardware and input.
- false (default): disables deterministic computation.
- true: enables deterministic computation for reduction communication operators.
- For the
Atlas A2 training product /Atlas A2 inference product , the supported communication operators include AllReduce, ReduceScatter, ReduceScatterV, and Reduce. - For the
Atlas A3 training product /Atlas A3 inference product , if the communication operator expansion mode is set to AI_CPU, all reduction operators run in deterministic computation mode and are not affected by this environment variable. If the communication operator expansion mode is set to Vector Core, only the AllReduce and ReduceScatter communication operators involve non-deterministic computation, and switching to deterministic computation is available after the variable is set to true.
- For the
- strict: enables strict deterministic computation for reduction communication operators, that is, the order-preserving function. This ensures that the reduction order of all bits is consistent on the basis of deterministic computation. The following conditions must be met to configure this value:
- Only symmetric multi-server distribution scenarios are supported, and asymmetric distribution scenarios (asymmetric card counts) are not supported.
- Only INF and NaN modes are supported, and the saturation mode is not supported.
- Compared with deterministic computation, enabling order preserving will cause performance deterioration. You are advised to use this function in inference scenarios.
- For the
Atlas A2 training product /Atlas A2 inference product , the supported communication operators include AllReduce, ReduceScatter, and ReduceScatterV. - For the
Atlas A3 training product /Atlas A3 inference product :- The supported communication operators include AllReduce and ReduceScatter. The supported data types are float16, float32, and bfp16. Only the sum operation is supported for reduction.
- The communication scale requires a rank size of no less than 3.
- For supernodes with multiple AI Servers, only SDMA communication over HCCS links between AI Servers is supported. RDMA communication over RoCE links is not supported in this case. Therefore, setting HCCL_INTER_HCCS_DISABLE to TRUE is not supported.
Generally, deterministic computation for reduction operators is disabled. You can enable it via this environment variable to aid debugging when inconsistent results occur across model runs or when investigating accuracy issues. Note that enabling this feature increases operator execution latency and degrades overall performance.
If deterministic computation for operators is enabled via this environment variable and the operator expansion mode is set to AIV (see HCCL_OP_EXPANSION_MODE for details), deterministic computation takes higher priority, and the AIV expansion may fail to take effect in certain scenarios.
Example
export HCCL_DETERMINISTIC=true
Constraints
If you call the HCCL C APIs to initialize a communicator with specific configurations and set the deterministic computation function using the hcclDeterministic parameter in HcclCommConfig, the configuration at the communicator granularity takes precedence.