Configuring Fault Handling Policies

Configuring Job-Level Rescheduling

Job-level Rescheduling is enabled by default. You only need to complete the steps for building an image and preparing the job YAML. For details about the feature introduction, usage constraints, supported product models, and principles of Job-level rescheduling, see Job-Level Rescheduling.

Preparing the Job YAML

In the job YAML, add the following fields to enable Job-level rescheduling.

...
metadata:
   labels:
     ...
     fault-scheduling: "force"

Configuring Pod-Level Rescheduling

This part guides you through the key steps for configuring Pod-level rescheduling. For details on the features, usage constraints, supported product models, and principles of Pod-level rescheduling, see Pod-Level Rescheduling.

Building an Image

Use a Dockerfile to build a container image and add a startup command. An example is shown below.

# Adaptation script to MindCluster resumable training. TASKD_WHL is the path to the TaskD whl installation package. Fill in the actual path accordingly.
# Optional. Under the PyTorch framework, the following commands must be configured when using graceful fault tolerance, Pod-level rescheduling, or process-level rescheduling.
RUN pip install $TASKD_WHL
RUN sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py

# Optional. Under the MindSpore framework, the following commands must be configured when using Pod-level rescheduling.
RUN pip install $TASKD_WHL

Preparing the Job YAML

In the job YAML, add the following fields to enable Pod-level rescheduling, modify the container port, and add port 9601 for TaskD communication under all Pods.

...
metadata:
   labels:
     ...
     pod-rescheduling: "on"
     fault-scheduling: "force"   # You can choose force or grace based on the actual situation. When configured as force, the Pod cannot use the host network.
...
        spec:
...
           containers:
...
             ports:
               - containerPort: 9601
                 name: taskd-port
...

Adapting the Training Script

In the startup script (for example, train_start.sh), add the following bold fields as shown in the example below.

 ...
 export MS_ENABLE_TFT="{RSC:1}"    # Configure this field to enable Pod-level rescheduling in MindSpore scenarios
 ...
 # Optional. In PyTorch scenarios, set the number of restarts within the container and the training process monitoring interval.
    logger "server id is: ""${server_id}"
    if [ "${framework}" == "PyTorch" ]; then
      get_env_for_pytorch_multi_node_job
      DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT --max_restarts 32767"

Where, --max_restarts specifies the maximum number of fault triggers allowed within the container, expressed as an integer. If this limit is exceeded, the PyTorch training process will exit immediately. If this parameter is not configured, the default value is 32767.

After the distributed environment initialization is complete and the global rank is obtained, modify the training script to start TaskD Manager in the training script.
1. Create a manager.py file and place it in the current directory when calling the training script. The content of the manager.py file is as follows.
```
from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX          # Total number of task nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()
```
  For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.
2. Add the following code to the training script (for example, train_start.sh) to start TaskD Manager. In the following code:
  - The two statements TASKD_SO_PATH and export LD_PRELOAD are used to configure the path of libtaskd.so (from the TaskD installation) into the environment variable LD_PRELOAD. If these two statements are not configured successfully, you can manually run the pip show taskd command to obtain the Location value, append /taskd/python/cython_api/libs/libtaskd.so, and then set it via export.
  - TASKD_PROCESS_ENABLE configuration instructions: If recover-strategy in the job YAML does not configure a recovery policy and does not enable hot switching, you need to configure export TASKD_PROCESS_ENABLE="off"; if recover-strategy is configured or hot switching is enabled, you do not need to configure export TASKD\_PROCESS\_ENABLE="off.
```
TASKD_SO_PATH="$(pip show taskd | awk '/^Location: / {print $2"/taskd/python/cython_api/libs/libtaskd.so"}')"
export LD_PRELOAD=$TASKD_SO_PATH:$LD_PRELOAD
export TASKD_PROCESS_ENABLE="off"
# Under the PyTorch framework
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path. The error.log log path must be created in advance.
fi
# Under the MindSpore framework
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path. The error.log log path must be created in advance.
fi
```

Configuring Process-Level Rescheduling

This part describes the key steps for configuring process-level rescheduling. For details about the feature introduction, usage constraints, supported product models, and principles of process-level rescheduling, see Process-Level Rescheduling.

Building an Image

Use a Dockerfile to build a container image and add a startup command.

# Adaptation script to MindCluster lossless resumable training. TASKD_WHL is the path to the TaskD whl installation package, and MINDIO_TTP_PKG is the path to the MindIO whl installation package. Fill them in according to the actual situation.
# Optional. Under the PyTorch framework, the following commands must be configured when using graceful fault tolerance, Pod-level rescheduling, or process-level rescheduling.
RUN pip3 install $TASKD_WHL
RUN pip3 install $MINDIO_TTP_PKG
RUN sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py

# Optional. Under the MindSpore framework, the following commands must be configured when using process-level rescheduling.
RUN pip3 install $MINDIO_TTP_PKG
RUN pip3 install $TASKD_WHL

Preparing the Job YAML

In the job YAML, modify the container port and add port 9601 for TaskD communication under all Pods.

...
        spec:
...
           containers:
...
             ports:
               - containerPort: 9601
                 name: taskd-port
...

In the job YAML, add the following fields to enable process-level rescheduling. recover-strategy is the strategy used for training process recovery, where recover indicates enabling process-level recovery.

Currently, process-level rescheduling supports the following two scenarios. Choose one based on the actual usage scenario.

Scenario 1: Migrate the faulty Pod to a healthy node after a fault occurs

  ...
  metadata:
     labels:
       ...
       fault-scheduling: "grace"
   ...
  ...
     annotations:
       ...
       recover-strategy: "recover"   # Recovery strategies (retry: process-level online recovery; recover: process-level rescheduling; recover-in-place: process-level in-place recovery; elastic-training: elastic training; dump: save dying gasps; exit: exit training). Six strategies can be combined arbitrarily, separated by commas.
   ...
  ...
  spec:
    replicaSpecs:
      Master:
        template:
          spec:
            containers:
            - name: ascend       # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
      Worker:
        template:
          spec:
            containers:
            - name: ascend # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
  ...

Scenario 2: Do not migrate the faulty Pod after a fault; only restart the faulty process

  ...
  metadata:
     labels:
       ...
       fault-scheduling: "grace"
   ...
  ...
     annotations:
       ...
       recover-strategy: "recover-in-place"   # Recovery strategies (retry: process-level online recovery; recover: process-level rescheduling; recover-in-place: process-level in-place recovery; elastic-training: elastic training; dump: save last words; exit: exit training). Six strategies can be combined arbitrarily, separated by commas.
   ...
  ...
  spec:
    replicaSpecs:
      Master:
        template:
          spec:
            containers:
            - name: ascend # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
      Worker:
        template:
          spec:
            containers:
            - name: ascend # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
  ...

Adapting the Training Script

(Optional) In the startup script (for example, train_start.sh), configure the --max_restarts parameter. An example is shown below.

 # In PyTorch scenarios, set the training process monitoring interval.
 ...
    logger "server id is: ""${server_id}"
    if [ "${framework}" == "PyTorch" ]; then
      get_env_for_pytorch_multi_node_job
      DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT  --max_restarts 32767"
 ...

Here, --max_restarts specifies the maximum number of fault triggers allowed within the container, expressed as an integer. If this limit is exceeded, the PyTorch training process will exit immediately. If this parameter is not configured, the default value is 32767.

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager in the training script.
1. Create a manager.py file in the current directory where the training script is invoked. The content of the manager.py file is as follows.
```
from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX          # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()
```
  For details about the parameters in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script (for example, train_start.sh) to start TaskD Manager. In the following code, the two statements TASKD_SO_PATH and export LD_PRELOAD configure the path of libtaskd.so (from the TaskD installation) into the environment variable LD_PRELOAD. If these two statements fail to configure successfully, you can manually run the pip show taskd command to obtain the value of Location, append /taskd/python/cython_api/libs/libtaskd.so to it, and then set it via export.

 ```shell
 TASKD_SO_PATH="$(pip show taskd | awk '/^Location: / {print $2"/taskd/python/cython_api/libs/libtaskd.so"}')"
 export LD_PRELOAD=$TASKD_SO_PATH:$LD_PRELOAD
 export TASKD_PROCESS_ENABLE="on"
 # For PyTorch Framework
 if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of `manager.py` is determined by the current path, and the `error.log` path must be created in advance.
 fi
 # For MindSpore Framework
 if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of `manager.py` is determined by the current path, and the `error.log` path must be created in advance.
 fi
 ```

Configuring Process-Level Online Recovery

This part describes the key steps for configuring process-level online recovery. For details about the features, usage constraints, supported product models, and principles of process-level online recovery, see Process-Level Online Recovery.

Build Image

Use a Dockerfile to build a container image and add a startup command.

# Adaptation script to MindCluster resumable training. TASKD_WHL is the path to the TaskD whl installation package, and MINDIO_TTP_PKG is the path to the MindIO whl installation package. Fill them in according to the actual situation.
# Optional. Under the PyTorch framework, you must configure the following commands when using graceful fault tolerance, Pod-level rescheduling, process-level rescheduling, or process-level online recovery.
RUN pip3 install $TASKD_WHL
RUN pip3 install $MINDIO_TTP_PKG
RUN sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py

# Optional. Under the MindSpore framework, you must configure the following commands when using process-level online recovery.
RUN pip3 install $TASKD_WHL
RUN pip3 install $MINDIO_TTP_PKG

Preparing the Job YAML

In the job YAML, add the following bold fields to enable process-level recovery, modify the container port, and add port 9601 for TaskD communication under all Pods.

...
   labels:
     ...
     fault-scheduling: "grace"
 ...
...
   annotations:
     ...
     recover-strategy: "retry"    # Recovery strategies (retry: process-level online recovery; recover: process-level rescheduling; recover-in-place: process-level in-place recovery; elastic-training: elastic training; dump: save last words; exit: exit training). Six strategies can be combined arbitrarily, separated by commas.
 ...
...
spec:
  replicaSpecs:
    Master:
      template:
        spec:
          containers:
          - name: ascend # Do not modify
            ...
            args:
              - |
                ...
                bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                  ...
            ports:
               - containerPort: 9601
                 name: taskd-port
...
    Worker:
      template:
        spec:
          containers:
          - name: ascend # Do not modify
            ...
            args:
              - |
                ...
                bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                  ...
            ports:
               - containerPort: 9601
                 name: taskd-port
...

In the MindSpore scenario, you need to modify the model parameter configuration YAML. Open the QWEN3_for_MS_code/configs/qwen3/pretrain_qwen3_32b_4k.yaml file and add the following bold fields.

# mindspore context init config
context:
  mode: 0  #0--Graph Mode; 1-Pynative Mode
  device_target: "Ascend"
  graph_kernel_flags: "--disable_pass=cluster.floatstatus_fusion,preprocess.depend_elimination"
  max_call_depth: 10000
  max_device_memory: "59GB"
  mempool_block_size: "59GB"
  save_graphs: True
  save_graphs_path: "./graph"
  device_id: 0
  jit_config:
    jit_level: "O1"
  memory_optimize_level: "00"
  ascend_config:
    hccl_watchdog: False

Adapting the Training Script

After the distributed environment initialization is complete and the global rank is obtained, modify the training script to start the TaskD Manager within the training script.

Create a manager.py file in the current directory where the training script is invoked. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX          # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script (for example, train_start.sh) to start TaskD Manager. In the following code, the two statements TASKD_SO_PATH and export LD_PRELOAD are used to configure the path of libtaskd.so (from TaskD installation) into the environment variable LD_PRELOAD. If these two statements fail to configure successfully, you can manually run the pip show taskd command to get the Location value, append /taskd/python/cython_api/libs/libtaskd.so, and then set it via export.

TASKD_SO_PATH="$(pip show taskd | awk '/^Location: / {print $2"/taskd/python/cython_api/libs/libtaskd.so"}')"
export LD_PRELOAD=$TASKD_SO_PATH:$LD_PRELOAD
export TASKD_PROCESS_ENABLE="on"
# Under the PyTorch framework
if [[ "${RANK}" == 0 ]]; then
   export MASTER_ADDR=${POD_IP}
   python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path, and the error.log log path must be created in advance.
fi
# Under the MindSpore framework
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
   python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path, and the error.log log path must be created in advance.
fi

(Optional) In the startup script (for example, train_start.sh), add the --max_restarts parameter. An example is shown below.

 ...
    logger "server id is: ""${server_id}"
    if [ "${framework}" == "PyTorch" ]; then
      get_env_for_pytorch_multi_node_job
      DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT --max_restarts 32767"

In the MindSpeed scenario, you need to modify the train_start.sh script and add the following fields in the code. An example is shown below.

  ```shell
  export HCCL_OP_RETRY_ENABLE="L0:0, L1:1, L2:1"   # Enable the re-execution feature of HCCL operators (operator-level online recovery). Re-execution means that when an SDMA or RDMA CQE type error is reported during the execution of a communication operator, HCCL will attempt to re-execute this communication operator.
  export HCCL_ASYNC_ERROR_HANDLING=0
  ```

In the MindFormers scenario, you need to modify the msrun_launcher.sh script and add the following fields in the code. An example is shown below.

  ```shell
  export HCCL_OP_RETRY_ENABLE="L0:0, L1:1, L2:1"  # This environment variable is used to configure whether to enable the re-execution feature of HCCL operators. Re-execution means that when an SDMA or RDMA CQE type error is reported during the execution of a communication operator, HCCL will attempt to re-execute this communication operator.
  ```

To test the process-level online recovery feature, configure it by referring to Process-level Online Recovery Verification.

Configuring Operator-level Online Recovery

This part describes key steps for configuring operator-level online recovery. For details on the feature introduction, usage constraints, supported product models, and principles of operator-level online recovery, see Operator-level Online Recovery.

Configuring Environment Variables

Before using operator-level online recovery, configure the environment variables HCCL_OP_RETRY_ENABLE and HCCL_OP_RETRY_PARAMS in the training startup script. For detailed descriptions of these environment variables, see the CANN Environment Variable Reference. A configuration example is shown below.

export HCCL_OP_RETRY_ENABLE="L0:0, L1:1, L2:1"     # Whether to enable the re-execution feature of HCCL operators
export HCCL_OP_RETRY_PARAMS="MaxCnt:3, HoldTime:5000, IntervalTime:1000"    # Configures specific parameters for HCCL operator re-execution, including the maximum number of re-execution attempts, the wait time before the first re-execution, and the interval between two re-executions

Configuring Suspension and Switchback for Link Failover Communication

PyTorch Scenario (Based on MindSpeed-LLM)

This section describes how to configure suspension and switchback of link failover communication. For details about its features, restrictions, supported products, and working principles, see Suspension and Switchback for Link Failover Communication.

Prerequisites

Complete the installation of the following components on the corresponding nodes: Ascend Docker Runtime, Ascend Operator, ClusterD, Ascend Device Plugin, and Volcano (The versions of the above MindCluster components must be compatible with TaskD.)
Install the following components in the container: torch_npu (7.1.RC1 or later), CANN (8.2.RC1 or later), TaskD, and MindIO (7.1.RC1 or later)

Procedure

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager in the script and start TaskD Worker within the training process.

Start TaskD Manager.

Create a manager.py file in the current directory where the training script is invoked. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py
export TASKD_PROCESS_ENABLE="on"
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi

torchrun ...

Start TaskD Worker.

Modify the QWEN3_for_PyTorch_2.7_code/mindspeed_llm/training/training.py file and add the following bold fields.

 def pretrain(train_valid_test_dataset_provider,
              model_provider,
              model_type,
              forward_step_func,
              process_non_loss_data_func=None,
              extra_args_provider=None,
              args_defaults={}):
     print_rank_0('time to initialize megatron (seconds): {:.3f}'.format(
         time.time() - _TRAIN_START_TIME))
     print_datetime('after megatron is initialized')
     import torch.distributed as dist
     if dist.is_initialized():
        rank = dist.get_rank()
        from taskd.api.taskd_worker_api import init_taskd_worker
        from taskd.api.taskd_worker_api import start_taskd_worker
        init_taskd_worker(rank,5000,"pt")
        start_taskd_worker()
     app_metrics['app_model_init_finish_time'] = one_logger_utils.get_timestamp_in_ms()
     one_logger_utils.on_pretrain_start()

If the error "the libtaskd.so has not been loaded" occurs during training, you need to import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload specified .so files. An example is shown below.

export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/lib/python3.10/dist-packages/taskd/python/cython_api/libs/libtaskd.so

libmspti.so: This .so file is provided by MindStudio and integrated within the CANN package. Default installation path: /usr/local/Ascend/cann/lib64/libmspti.so.
libtaskd.so: This .so file is provided by TaskD, and the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so. The TaskD installation path can be queried using the following command. The Location field in the output is the TaskD installation path.
```
pip show taskd
```

Modify the training framework code.
1. Go to the "mindcluster-deploy" repository, switch to the corresponding version branch according to mindcluster-deploy Open Source Repository Version Description, obtain the train_start.sh file from the samples/train/resumable-training/fault-tolerance/without-ranktable/pytorch/Qwen3 directory, and construct the following directory structure on the management node.
```
root@ubuntu:/data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/scripts#
scripts/
└── train_start.sh
```
2. Configure the training startup script train_start.sh, and add the following fields to the code.
```
# Enable the re-execution feature for HCCL operators. Re-execution means that when SDMA or RDMA CQE type errors are reported during the execution of a communication operator, HCCL will attempt to re-execute this communication operator.
export HCCL_OP_RETRY_ENABLE="L0:0, L1:1, L2:1"
```

Modify the job YAML.

Add the following bold fields to the job YAML to enable process-level online recovery, and modify the container port by adding port 9601 for TaskD communication under all Pods.

 ...
     labels:
       ...
       fault-scheduling: "grace"
    ...
 ...
     annotations:
       ...
       recover-strategy: "retry"    # Recovery strategy. The value retry indicates that process-level online recovery is enabled.
    ...
 ...
 spec:
    replicaSpecs:
      Master:
        template:
          spec:
            containers:
            - name: ascend # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
              ports: 
                - containerPort: 9601
                  name: taskd-port
 ...
      Worker:
        template:
          spec:
            containers:
            - name: ascend # Do not modify
              ...
              args:
                - |
                  ...
                  bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                    ...
              ports:
                - containerPort: 9601
                  name: taskd-port
 ...

MindSpore Scenario (Based on MindFormers)

Prerequisites

Install the following components on the corresponding nodes: Ascend Docker Runtime, Ascend Operator, ClusterD, Ascend Device Plugin, and Volcano (the versions of the above MindCluster components must be compatible with TaskD)
Install the following components in the container: torch_npu (7.1.RC1 or later), CANN (8.2.RC1 or later), TaskD, and MindIO (7.1.RC1 or later)

Procedure

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager in the training script, start TaskD Proxy in the management process, and start TaskD Worker in the training process.

Start TaskD Manager.

Create a manager.py file in the current directory when calling the training script. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For details about the parameters in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

export TASKD_PROCESS_ENABLE="on"
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi

msrun ...

Start TaskD Worker. Modify the ./mindformers/trainer/base_trainer.py file and add the following bold fields.

     def training_process(
             self,
             config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None,
             network: Optional[Union[Cell, PreTrainedModel]] = None,
             dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None,
             optimizer: Optional[Optimizer] = None,
             callbacks: Optional[Union[Callback, List[Callback]]] = None,
             compute_metrics: Optional[Union[dict, set]] = None,
             **kwargs):
         ……
         ……

     logger.info(".........Starting Training Model..........")
     if get_real_rank() % 8 == 0:
         pprint(config)
     logger.info(".........Model Compiling, Please Wait a Moment...........")
     <strong>try:</strong>
         <strong>rank = get_rank()</strong>
         <strong>from taskd.api.taskd_worker_api import init_taskd_worker</strong>
         <strong>from taskd.api.taskd_worker_api import start_taskd_worker</strong>
         <strong>init_taskd_worker(rank,5000,"ms")</strong>
         <strong>start_taskd_worker()</strong>
     <strong>except Exception as e:</strong>
         <strong>print("failed to call mindcluster taskd")</strong>
     model.train(config.runner_config.epochs, dataset,
                 callbacks=callbacks,
                 dataset_sink_mode=config.runner_config.sink_mode,
                 sink_size=config.runner_config.sink_size,
                 initial_epoch=config.runner_config.initial_epoch)</pre>

Modify the training framework code to enable the track borrowing switch.

Edit QWEN3_for_MS_code/scripts/msrun_launcher.sh and add the following fields to the code.
```
export MS_ENABLE_TFT="{TTP:1,TSP:1}"           # Enable dying gasp and link failover.
export HCCL_OP_RETRY_ENABLE="L0:0, L1:1, L2:1"  # This environment variable is used to configure whether to enable the re-execution feature of HCCL operators. Re-execution means that when an SDMA or RDMA CQE type error is reported during the execution of a communication operator, HCCL will attempt to re-execute this communication operator.
```
If the error "the libtaskd.so has not been loaded" occurs during training, you need to import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload the specified so file. An example is as follows.
```
export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so
```
- libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. The default installation path is /usr/local/Ascend/cann/lib64/libmspti.so.
- libtaskd.so: This .so file is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so. You can run the following command to query the path where TaskD is located. The Location field in the command output is the target path.
```
pip show taskd
```

Modify the job YAML.

Add the following bold fields in the job YAML to enable process-level online recovery, and modify the container port by adding port 9601 for TaskD communication under all Pods.

 ...
     labels:
       ...
       fault-scheduling: "grace"
   ...
 ...
     annotations:
       ...
       recover-strategy: "retry"    # Recovery strategy. The value retry indicates enabling process-level online recovery.
   ...
 ...
 spec:
   replicaSpecs:
     Master:
       template:
         spec:
           containers:
           - name: ascend # Do not modify
             ...
             args:
               - |
                 ...
                 bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                   ...
             ports:
               - containerPort: 9601
                 name: taskd-port
 ...
     Worker:
       template:
         spec:
           containers:
           - name: ascend # Do not modify
             ...
             args:
               - |
                 ...
                 bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                   ...
             ports:
               - containerPort: 9601
                 name: taskd-port
 ...

Configuring Graceful Fault Tolerance

This function has been deprecated. It will not be supported in PyTorch versions beyond 7.2.RC1 and MindSpore versions beyond 7.1.RC1.

This section describes how to configure graceful fault tolerance. For details about its features, restrictions, supported products, and working principles, see (Optional) Graceful Fault Tolerance.

Building an Image

Use a Dockerfile to build a container image and add a startup command.

# Adaptation script to MindCluster resuamble training. MINDIO_TTP_PKG is the path to the MindIO whl installation package. Fill it in according to the actual situation.
RUN pip3 install $MINDIO_TTP_PKG

Adapting the Training Script

In the startup script (for example, train_start.sh), add the following fields. An example is shown below.

...
export MS_ENABLE_TFT="{RSC:1}"      # Configure this field in MindSpore scenarios to enable graceful fault tolerance.
...

Configuring the Startup YAML File

Modify the startup YAML of Ascend Device Plugin, set -hotReset=1 to enable hot reset, and use graceful fault tolerance mode. Note: Graceful fault tolerance cannot be enabled simultaneously with process-level rescheduling or process-level online recovery.

...
      containers:
      - image: ascend-k8sdeviceplugin:v{version}
        name: device-plugin-01
        resources:
          requests:
            memory: 500Mi
            cpu: 500m
          limits:
            memory: 500Mi
            cpu: 500m
        command: [ "/bin/bash", "-c", "--"]
        args: [ "device-plugin
                 -useAscendDocker=true
                 -volcanoType=true                    # Volcano must be used in rescheduling scenarios.
                 -autoStowing=true                    # Whether to enable automatic management. The default value is true. If this parameter is set to false, automatic management is disabled. In this case, after the processor health status changes from unhealthy to healthy, or the network fault on the processor parameter plane is recovered, the processor will not be automatically added to the schedulable resource pool. This parameter applies only to Atlas training products.
                 -listWatchPeriod=5                   # Sets the health status check period, in seconds. Range: [3,1800]
                 -hotReset=1      # Enable the hot reset function and use graceful fault tolerance mode on top of Job-level or Pod-level rescheduling.
                 -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log
                 -logLevel=0" ]
        securityContext:
          privileged: true
          readOnlyRootFilesystem: true
...

Configuring Online Stress Testing

PyTorch Scenario (Based on MindSpeed-LLM)

This section guides users through the key steps for configuring online stress testing. For details on the feature introduction, usage constraints, and supported product models of online stress testing, see Online Stress Testing.

Prerequisites

Complete the installation of the following components on the corresponding nodes: Ascend Docker Runtime, Ascend Operator, ClusterD, Ascend Device Plugin, and Volcano (The versions of the above MindCluster components must be compatible with TaskD.)
Install the following components in the container: torch_npu (7.1.RC1 or later), CANN (8.2.RC1 or later), TaskD, and MindIO (7.1.RC1 or later)

Procedure

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager within the training script and start TaskD Worker inside the training process.

Start the TaskD Manager.

Create a manager.py file and place it in the current directory where the training script is invoked. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         #Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For details about the parameters in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py
export TASKD_PROCESS_ENABLE="on"
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi

torchrun ...

Start TaskD Worker.

Modify the QWEN3_for_PyTorch_2.7_code/mindspeed_llm/training/training.py file and add the following bold fields.

 def pretrain(train_valid_test_dataset_provider,
              model_provider,
              model_type,
              forward_step_func,
              process_non_loss_data_func=None,
              extra_args_provider=None,
              args_defaults={}):
     print_rank_0('time to initialize megatron (seconds): {:.3f}'.format(
         time.time() - _TRAIN_START_TIME))
     print_datetime('after megatron is initialized')
     import torch.distributed as dist
     if dist.is_initialized():
        rank = dist.get_rank()
        from taskd.api.taskd_worker_api import init_taskd_worker
        from taskd.api.taskd_worker_api import start_taskd_worker
        init_taskd_worker(rank,5000,"pt")
        start_taskd_worker()
     app_metrics['app_model_init_finish_time'] = one_logger_utils.get_timestamp_in_ms()
     one_logger_utils.on_pretrain_start()

export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/lib/python3.10/dist-packages/taskd/python/cython_api/libs/libtaskd.so

libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. The default installation path is /usr/local/Ascend/cann/lib64/libmspti.so.
libtaskd.so: This so is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so. You can run the following command to query the path where TaskD is located. The Location field in the command output is the target path.
```
pip show taskd
```

Modify the job YAML.

Add the following bold fields in the job YAML to enable process-level rescheduling and add port 9601 for TaskD communication under all Pods.

     ...
        labels:
          ...
          fault-scheduling: "grace"
      ...
     ...
        annotations:
          ...
          recover-strategy: "recover"    # Recovery strategy. The value recover indicates enabling process-level rescheduling.
      ...
     ...
     spec:
       replicaSpecs:
         Master:
           template:
             spec:
               containers:
               - name: ascend # do not modify
                 ...
                 args:
                   - |
                     cd /job/code;
                     chmod +x scripts/train_start.sh;
                     bash scripts/train_start.sh
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...
         Worker:
           template:
             spec:
               containers:
               - name: ascend # do not modify
                 ...
                 args:
                   - |
                     cd /job/code;
                     chmod +x scripts/train_start.sh;
                     bash scripts/train_start.sh
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...

MindSpore Scenario (Based on MindFormers)

This section describes how to configure online stress testing. For details about its features, restrictions, supported products, and working principles, see Online Stress Testing.

Prerequisites

Install the following components on the corresponding nodes: Ascend Docker Runtime, Ascend Operator, ClusterD, Ascend Device Plugin, and Volcano (the versions of the above MindCluster components must be compatible with TaskD)
Install MindSpore (version 2.7.0 or later), CANN (version 8.2.RC1 or later), TaskD, and MindIO (version 7.2.RC1 or later) in the container.

Procedure

Start TaskD Manager.

Create a manager.py file in the current directory where the training script is invoked. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

export TASKD_PROCESS_ENABLE="on"
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi

msrun ...

Start TaskD Worker. Modify the ./mindformers/trainer/base_trainer.py file and add the following bold fields.

     def training_process(
             self,
             config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None,
             network: Optional[Union[Cell, PreTrainedModel]] = None,
             dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None,
             optimizer: Optional[Optimizer] = None,
             callbacks: Optional[Union[Callback, List[Callback]]] = None,
             compute_metrics: Optional[Union[dict, set]] = None,
             **kwargs):
         ……
         ……

         logger.info(".........Starting Training Model..........")
         if get_real_rank() % 8 == 0:
             pprint(config)
         logger.info(".........Model Compiling, Please Wait a Moment...........")
         try:
             rank = get_rank()
             from taskd.api.taskd_worker_api import init_taskd_worker
             from taskd.api.taskd_worker_api import start_taskd_worker
             init_taskd_worker(rank,5000,"ms")
             start_taskd_worker()
         except Exception as e:
             print("failed to call mindcluster taskd")
         model.train(config.runner_config.epochs, dataset,
                     callbacks=callbacks,
                     dataset_sink_mode=config.runner_config.sink_mode,
                     sink_size=config.runner_config.sink_size,
                     initial_epoch=config.runner_config.initial_epoch)

Modify the training framework code and enable online stress testing.

Edit the startup script QWEN3_for_MS_code/scripts/msrun_launcher.sh file and add the following fields to the code.
```
export MS_ENABLE_TFT="{TTP:1,TSP:1}"           # Enable dying gasp and online stress testing.
```
If the error "the libtaskd.so has not been loaded" occurs during training, you need to import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload the specified .so file. An example is as follows.
```
export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so
```
- libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. The default installation path is /usr/local/Ascend/cann/lib64/libmspti.so.
- libtaskd.so: This .so file is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so. You can run the following command to query the path where TaskD is located. The Location field in the command output is the target path.
```
pip show taskd
```

Modify the job YAML.

Add the following bold fields to the job YAML to enable process-level rescheduling, and modify the container port by adding port 9601 for TaskD communication under all Pods.

     ...
        labels:
          ...
          fault-scheduling: "grace"
      ...
     ...
        annotations:
          ...
          recover-strategy: "recover"    # Recovery policy. The value is recover, indicating that process-level rescheduling is enabled.
      ...
     ...
     spec:
       replicaSpecs:
         Master:
           template:
             spec:
               containers:
               - name: ascend # Do not modify
                 ...
                 command:                           # Training command, which can be modified
                   - /bin/bash
                   - -c
                   - |
                    cd /job/code/;bash scripts/msrun_launcher.sh "run_mindformer.py --config configs/qwen3/pretrain_qwen3_32b_4k.yaml --auto_trans_ckpt False --use_parallel True --run_mode train"
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...
         Worker:
           template:
             spec:
               containers:
               - name: ascend # Do not modify
                 ...
                 command:                           # Training command, which can be modified
                   - /bin/bash
                   - -c
                   - |
                    cd /job/code/;bash scripts/msrun_launcher.sh "run_mindformer.py --config configs/qwen3/pretrain_qwen3_32b_4k.yaml --auto_trans_ckpt False --use_parallel True --run_mode train"
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...

Configuring Hot Switching

This section describes how to configure hot switching. For details about its features, restrictions, supported products, and working principles, see Hot Switching.

Building an Image

Use a Dockerfile to build a container image and add a startup command. An example is shown below.

# Adaptation script to MindCluster resumable training. TASKD_WHL is the path to the TaskD whl installation package, MINDIO_TTP_PKG is the path to the MindIO whl installation package, and MINDSPORE_WHL is the path to the MindSpore whl installation package. Please fill in the paths according to your actual situation.
# Optional. Under the PyTorch framework, you must configure the following command when using hot switching.
RUN pip3 install $TASKD_WHL
RUN pip3 install $MINDIO_TTP_PKG
RUN sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py

# Optional. Under the MindSpore framework, you must configure the following command when using hot switching.
RUN pip3 install $MINDIO_TTP_PKG
RUN pip3 install $TASKD_WHL
RUN pip3 install $MINDSPORE_WHL

Prepararing the Job YAML

In the job YAML, add the following fields to enable hot switching, modify the container port, and add port 9601 for TaskD communication under all Pods.

...
metadata:
   labels:
     ...
     subHealthyStrategy: "hotSwitch"
...
        spec:
...
           containers:
...
             ports:
               - containerPort: 9601
                 name: taskd-port
...

Adapting the Training Script

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager within the training script.

Create a manager.py file and place it in the current directory when invoking the training script. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX          # Total number of nodes
proc_per_node=XX     # Training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

export TASKD_PROCESS_ENABLE="on"

# Under the PyTorch framework
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path. The error.log log path must be created in advance.
fi

torchrun ...

# Under the MindSpore framework
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path. The error.log log path must be created in advance.
fi

msrun ...

export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/lib/python3.10/dist-packages/taskd/python/cython_api/libs/libtaskd.so

libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. The default installation path is /usr/local/Ascend/cann/lib64/libmspti.so.
libtaskd.so: This .so file is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so. You can run the following command to query the path where TaskD is located. The Location field in the command output is the target path.
```
pip show taskd
```

Configuring Elastic Training

This section describes how to configure elastic training. For details about its features, restrictions, supported products, and working principles, see Elastic Training.

Prerequisites

Complete the installation of the following components on the corresponding nodes: Ascend Docker Runtime, Ascend Operator, ClusterD, Ascend Device Plugin, and Volcano (The versions of the above MindCluster components must be compatible with TaskD.)
Install the following components in the container: torch_npu (7.1.RC1 or later), CANN (8.2.RC1 or later), TaskD, and MindIO (7.1.RC1 or later)

Procedure

After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Manager in the training script.

Create a manager.py file in the current directory where the training script is invoked. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py
export TASKD_PROCESS_ENABLE="on"
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi

torchrun ...

Modify the job YAML.

Add the following bold fields to the job YAML to enable elastic training, and modify the container port by adding port 9601 for TaskD communication under all Pods.

     ...
        labels:
          ...
          fault-scheduling: "grace"
      ...
     ...
        annotations:
          ...
          wait-reschedule-timeout: "270" # Timeout for waiting for the faulty node to be rescheduled during process-level recovery. The default is 270 seconds, with a valid range of 30 to 270. When both process-level recovery and elastic training are enabled, if the faulty node is successfully scheduled within this time, process-level recovery is performed; otherwise, elastic training is triggered.
          recover-strategy: "elastic-training"    # Available recovery strategy. The value is elastic-training, indicating that elastic training is enabled.
      ...
     ...
     spec:
       replicaSpecs:
         Master:
           template:
             spec:
               containers:
               - name: ascend # do not modify
                 env:
                   - name: MINDIO_WAIT_MINDX_TIME         # It is recommended to configure this to 60 or above when process-level recovery is not enabled and elastic training is enabled.
                     value: "60"
                 args:
                   - |
                     ...
                     bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                       ...
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...
         Worker:
           template:
             spec:
               containers:
               - name: ascend # do not modify
                 env:
                   - name: MINDIO_WAIT_MINDX_TIME         # Recommended to set to 60 or above when process-level recovery is disabled and elastic training is enabled
                     value: "60"
                 args:
                   - |
                     ...
                     bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                       ...
                 ports:
                   - containerPort: 9601
                     name: taskd-port
     ...

Modify the training framework code.

Go to the "mindcluster-deploy" repository, switch to the corresponding version branch according to the mindcluster-deploy Open Source Repository Version Description, obtain the train_start.sh file from the samples/train/resumable-training/fault-tolerance/without-ranktable/pytorch/Qwen3 directory, and construct the following directory structure on the management node.
```
root@ubuntu:/data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/scripts#
scripts/
└── train_start.sh
```

Parameter Description

Different fault handling modes require different parameters to be configured, as shown in Table 1. For details about the meaning and filling instructions of each parameter, see Table 2.In scenarios such as process-level rescheduling, process-level online recovery, process-level in-place recovery, and elastic training, Ascend Operator injects different environment variables based on the user-configured recover-strategy and pod-rescheduling, and automatically labels the job with process-recover-enable=on to enable process-level recovery, without requiring manual specification by the user. The specific injected environment variables are shown in Table 3.

Table 1 Parameters required for fault handling

-	Job-Level Rescheduling	Pod-Level Rescheduling	Process-Level Rescheduling (recover)	Process-level In-place Recovery (recover-in-place)	Process-level Online Recovery	Graceful Fault Tolerance	Elastic Training
hotReset	-	-	-	-	-	√	-
fault-scheduling	√	√	√	√	√	-	√
pod-rescheduling	-	√	-	-	-	-	-
process-recover-enable	-	-	√	√	√	-	√
recover-strategy	-	-	√	√	√	-	√
PROCESS_RECOVER	-	-	√	√	√	-	√
ENABLE_RESTART_FAULT_PROCESS	-	-	-	√	-	-	-
ELASTIC_PROCESS_RECOVER_ENABLE	-	-	√	√	√	-	-
--enable-high-availability (MindSpeed-LLM side parameter)	-	-	√	√	√	-	√
--enable-hbmfault-repair (MindSpeed-LLM side parameter)	-	-	-	-	√	-	-
--enable-worker-reboot (MindSpeed-LLM side parameter)	-	-	√	√	-	-	-
--enable-elastic-training (MindSpeed-LLM side parameter)	-	-	-	-	-	-	√
max_restarts	-	√	√	√	√	-	-
monitor_interval	-	√	√	√	√	-	-
fault-retry-times	√	√	√	-	-	-	√

Table 2 Parameter description

Parameter Name	Parameter Location	Parameter Description
hotReset	Startup YAML of Ascend Device Plugin	Graceful fault tolerance switch. Value 1: When using resumable training, you can enable the hot reset feature on top of Job-level or Pod-level rescheduling to use graceful fault tolerance mode; Value 2: When using process-level recovery, set the hotReset parameter value to 2 to enable offline recovery mode. [!NOTE] NOTE The feature corresponding to value 1 has been sunset. Please configure other values.
pod-rescheduling	metadata.labels of the training job YAML	on: Enable Pod-level rescheduling. Other values or not using this field: Disable Pod-level rescheduling.
fault-scheduling	metadata.labels of the training job YAML	Rescheduling switch.
process-recover-enable	metadata.labels of the training job YAML	on: Enable process-level rescheduling and process-level online recovery. Process-level rescheduling and graceful fault tolerance cannot be enabled simultaneously. If both are enabled, checkpoint restart will resume training through Job-level rescheduling. pause: Temporarily disable process-level rescheduling and process-level online recovery. off or not using this field: Disable process-level rescheduling and process-level online recovery.
recover-strategy	metadata.annotations of the training job YAML	Available recovery strategies for the job. retry: Process-level online recovery. recover: Process-level rescheduling. recover-in-place: Process-level in-place recovery. elastic-training: Elastic training. dump: Save last words. exit: Exit training.
PROCESS_RECOVER	spec.replicaSpecs.{ Master \|Scheduler\| Worker}.template.spec.containers.env of the training job YAML	Master switch on the Elastic Agent/TaskD side for process-level rescheduling and process-level online recovery. on: Enable. off: Disable.
ELASTIC_PROCESS_RECOVER_ENABLE	spec.replicaSpecs.{ Master\|Scheduler\| Worker}. template.spec.containers.args of the startup training YAML	Switch on the Elastic Agent side for process-level rescheduling, process-level online recovery, and last CKPT recovery features. Value 1: Enable this feature. Other values: Disable this feature. When disabling this feature, the related features on the MindIO side must be disabled simultaneously. [!NOTE] NOTE The Elastic Agent component has been sunset, and related materials will be removed in the version released on December 30, 2026. This environment variable will be removed accordingly.
ENABLE_RESTART_FAULT_PROCESS	spec.replicaSpecs.{ Master\|Scheduler\| Worker}. template.spec.containers.args of the startup training YAML	Switch for the Elastic Agent/TaskD component to enable the in-place recovery feature for faulty processes. on: Enable this feature; Other values: Disable this feature
--enable-high-availability	Startup parameter of the training script pretrain_gpt.py	Fault fast recovery feature switch, disabled by default. When configured, the last words feature is enabled.
--enable-hbmfault-repair	Startup parameter of the training script pretrain_gpt.py	Process-level online recovery feature switch, disabled by default. When configured, fault detection is performed on on-chip memory and online repair is completed. Must be enabled together with enable-high-availability.
--enable-worker-reboot	Startup parameter of the training script pretrain_gpt.py	Process-level rescheduling feature switch, disabled by default. When configured, process-level scheduling is performed when a general fault occurs. Must be enabled together with enable-high-availability.
--enable-elastic-training	Startup parameter of the training script pretrain_gpt.py	Elastic training feature switch, disabled by default.
max_restarts	In the shell script for starting training (e.g., train_start.sh)	Configures the maximum number of fault triggers allowed within the container, with an integer value. If this number is exceeded, the PyTorch training process will exit training directly. The default value is 32767 if this parameter is not configured.
monitor_interval	In the shell script for starting training (e.g., train_start.sh)	Configures the time interval for monitoring the training process status, in seconds, with an integer value. The default value is 5 seconds if this parameter is not configured.
HIGH_AVAILABILITY	In the environment variables injected into the container by Ascend Operator	Ascend Operator automatically injects this environment variable based on the job type. When using MindSpeed-LLM version 2.3.0, this environment variable is automatically read, eliminating the need to manually add the --enable-high-availability, --enable-hbmfault-repair, --enable-worker-reboot, and --enable-elastic-training parameters in train_start.sh to enable the corresponding features.

Table 3 Environment variables injected by Ascend Operator

-	recover	retry	recover-in-place	elastic-training	dump	exit	pod-rescheduling
PyTorch	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 HIGH_AVAILABILITY=recover	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 HIGH_AVAILABILITY=retry	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 ENABLE_RESTART_FAULT_PROCESS=on HIGH_AVAILABILITY=recover	PROCESS_RECOVER=on HIGH_AVAILABILITY=elastic-training	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 HIGH_AVAILABILITY=dump	-	-
MindSpore	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 MINDIO_FOR_MINDSPORE=1 MS_ENABLE_TFT={ ARF:1}	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 MINDIO_FOR_MINDSPORE=1 MS_ENABLE_TFT={ UCE:1, HCCE:1}	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 ENABLE_RESTART_FAULT_PROCESS=on MINDIO_FOR_MINDSPORE=1 MS_ENABLE_TFT={ ARF:1}	-	PROCESS_RECOVER=on ELASTIC_PROCESS_RECOVER_ENABLE=1 MINDIO_FOR_MINDSPORE=1 MS_ENABLE_TFT={ TTP:1}	-	MS_ENABLE_TFT={ RSC:1}