Description
During distributed training or inference, tasks executed by different device processes may be inconsistent, for example, only specific processes save the checkpoint data. This environment variable controls the synchronization wait time during task execution between devices. Within this configured time, each device process waits for other devices to perform communication synchronization.
- For the Atlas 350 Accelerator Card: The value range is [0, 2147483647], in seconds. The default value is 1836. Only integer values are allowed. A value of 0 means no timeout limit.
- For the
Atlas A3 training product /Atlas A3 inference product :- In AI_CPU and AICPU_CacheDisable modes (see HCCL_OP_EXPANSION_MODE for details), the value range is [0, 2147483647], in seconds. The default value is 1836. Configurations with 10-millisecond precision are supported (for example, set it to 0.05 for a 50-millisecond timeout). A value of 0 means no timeout limit.
- In AIV mode, the value range is [0, 1091], in seconds. The default value is 1091. Configurations with 10-millisecond precision are supported (for example, set it to 0.05 for a 50-millisecond timeout). If the value is set to 0 or exceeds the maximum value 1091, the value 1091 is used.
In AIV mode, the actual effective timeout is interval × N × 10–3 milliseconds, where interval denotes the minimum operator timeout interval supported by the hardware (obtainable via the aclrtGetOpTimeoutInterval API), and N is an integer ranging from 1 to 254. If the configured timeout value does not equal interval × N × 10–3, the value will be rounded up to the nearest value of interval × N × 10–3.
- For the
Atlas A2 training product /Atlas A2 inference product :- In HOST and HOST_TS modes (see HCCL_OP_EXPANSION_MODE for details), the value range is [0, 2147483647], in seconds. The default value is 1836. Only integer values are allowed. A value of 0 means no timeout limit.
- In AIV mode, the value range is [0, 1091], in seconds. The default value is 1091. Configurations with 10-millisecond precision are supported (for example, set it to 0.05 for a 50-millisecond timeout). If the value is set to 0 or exceeds the maximum value 1091, the value 1091 is used.
In AIV mode, the actual effective timeout is interval × N × 10–3 milliseconds, where interval denotes the minimum operator timeout interval supported by the hardware (obtainable via the aclrtGetOpTimeoutInterval API), and N is an integer ranging from 1 to 254. If the configured timeout value does not equal interval × N × 10–3, the value will be rounded up to the nearest value of interval × N × 10–3.
- For the
Atlas training product : The value range is (0, 17340], in seconds. The default value is 1836. Only integer values are allowed.Note: For the
Atlas training product , the actual timeout set by the system is calculated as (value of the environment variable // 68) × 68 (unit: second). If the configured value of the environment variable is less than 68, the system will use 68 seconds as the timeout by default.For example, if HCCL_EXEC_TIMEOUT is set to 600, the actual timeout interval is 600 // 68 × 68 = 8 × 68 = 544s.
- For the
Atlas inference product : The value range is (0, 17340], in seconds. The default value is 1836. Only integer values are allowed.Note: For the
Atlas inference product , the actual timeout set by the system is calculated as (value of the environment variable // 68) × 68 (unit: second). If the configured value of the environment variable is less than 68, the system will use 68 seconds as the timeout by default.For example, if HCCL_EXEC_TIMEOUT is set to 600, the actual timeout interval is 600 // 68 × 68 = 8 × 68 = 544s.
In normal cases, you need to retain the default configuration. When the default value cannot meet the requirements for communication synchronization between devices, set this environment variable to increase the synchronization wait period between devices.
Example
export HCCL_EXEC_TIMEOUT=1800
Constraints
If you call the HCCL C APIs to initialize a communicator with specific configurations and set the synchronization waiting timeout for inter-device execution via the hcclExecTimeOut parameter in HcclCommConfig, the configuration at the communicator granularity takes precedence.
Applicability
Atlas 350 Accelerator Card