Solutions and Principles

Fault Detection

Fault Description

Resumable training identifies fault states in the cluster and training services using fault detection mechanisms and resolves issues based on the detection results. Currently, this feature supports fault detection for Ascend hardware faults, training service faults, and other faults.

Among MindCluster cluster scheduling components, Ascend Device Plugin detects NPU faults and NPU parameter plane network faults; NodeD detects server faults, DPC shared storage faults, and UnifiedBus network faults; ClusterD detects public faults; and Volcano detects container exceptions on the service plane. The figure below shows the overall fault detection architecture.

1. Ascend Device Plugin on the computing server obtains NPU chip faults and parameter plane network faults through the driver, and then reports the fault information to the management server.
2. NodeD on the computing server obtains server node faults, DPC shared storage faults, and UnifiedBus network fault information through the driver, and then reports the fault information to the management server.
3. Kubernetes on the computing server monitors the status of the training container. If an abnormality occurs, the fault is reported to Kubernetes, and Volcano, deployed on the management server, retrieves the fault information through Kubernetes.
4. After obtaining public faults through the public fault interface, ClusterD on the management server aggregates the received information and writes it into cluster-info-device-cm.
5. (Optional) ClusterD on the management server aggregates fault information reported by all Ascend Device Plugin and NodeD components within a cluster.

Supported Fault Types

Currently, more than 200 faults can be detected. For details about the fault types, see Table 1.For detailed fault descriptions, see Typical Faults.

Table 1 Fault types

Status:       Failed

ConfigMap Description

• Ascend Device Plugin on each computing node creates a ConfigMap file that records the NPU and UnifiedBus device information of the node. This ConfigMap file is named mindx-dl-deviceinfo-_<nodename>_ (hereinafter referred to as device-info-cm), and fault information is reported through this ConfigMap. For descriptions of the fields in this ConfigMap file, see the DeviceInfoCfg table.

• When a node fault exists on a node, NodeD on each computing node creates a ConfigMap file that records the device information of the node. This ConfigMap file is named mindx-dl-nodeinfo-_<nodename>_ (hereinafter referred to as node-info-cm), and node fault information is reported through this ConfigMap. For descriptions of the fields in this ConfigMap file, see the mindx-dl-nodeinfo-<nodename> table.

• ClusterD creates a ConfigMap file that records the cluster device information. The ConfigMap file is named cluster-info-<device/switch\>-<[0-5]> or cluster-info-node-cm (hereinafter referred to as cluster-info-cm). Node and chip fault information is reported via cluster-info-cm.

• When creating each job, you need to configure a ConfigMap file in the YAML. The ConfigMap file is named reset-config-_<job-name>_ (hereinafter referred to as reset-info-cm). This ConfigMap is mounted to the container's /user/restore/reset/config path. Ascend Device Plugin automatically mounts the ConfigMap to the /user/restore/reset/<job-namespace>.<job-name> path on the local node.

  You can also replace the ConfigMap with /user/restore/reset/<job-namespace>.<job-nam> on the node and mount it to the container's /user/restore/reset/config path. For field descriptions of this ConfigMap file, see the reset-config-<job-name> table.

Node Faults

Node fault discovery is primarily implemented through NodeD. Node faults include node health status, node hardware faults, and node DPC shared storage faults. Detailed descriptions are as follows:

• Node health status

  After completing the node status diagnosis of the current node, NodeD collects fault information within this node. When a node fault occurs, it continuously sends the node status to Volcano through the node status reporting mechanism (currently, only hardware fault information within this node is collected).

• Node hardware faults

  For node hardware faults, NodeD sends a fault query request to iBMC through the IPMI driver, and iBMC responds with the current hardware alarm information to NodeD. After collecting the hardware alarm information, NodeD reports the node hardware status to Volcano.

• Node DPC shared storage faults

For nodes using the Scale-Out Storage DPC product, you can start the NodeD service using the noded-dpc-{version}.yaml file from the NodeD installation package. This enables detection and reporting of DPC process exceptions and out-of-memory exceptions.

Figure 1 Node fault reporting

• When a node fault occurs, NodeD updates the node-info-cm content of the current node within a minimum of 5 seconds (default). For field descriptions, see the mindx-dl-nodeinfo-<nodename> table.
• NodeD queries fault information from iBMC every 60 seconds (default). When the queried fault information changes compared to the last query or the interval since the last report exceeds 30 minutes, it is reported to node-info-cm within 1 second.

Required Components

To ensure the proper functioning of the node fault detection feature, the following components must be installed: Volcano, Ascend Operator, NodeD, ClusterD

Constraints

• The node hardware fault reporting capability of NodeD only supports the following products: Atlas 800T A2 training server, Atlas 900 A2 PoD cluster basic unit, Atlas 900 A3 SuperPoD.
• Only iBMC versions V2 3.15.0.1 and later, or V2 3.10.02.55, with the IPMC driver installed, support the node hardware fault reporting capability of NodeD. If an earlier iBMC or IPMI version fails to obtain node fault information, only the node health status will be reported.
• To use the SuperPoD fault detection feature, iBMC V3 5.8.3.35 or later is required.
• To use the DPC fault detection feature, Scale-Out Storage DPC 24.2.0 or later is required.

Supported Fault Handling Types

Job-level rescheduling, Pod-level rescheduling, process-level rescheduling

(Optional) Configuring Fault Detection Levels

Resumable training provides default fault levels and fault handling policies for different fault codes of node hardware faults. If you want to modify the fault handling policy, see Node Hardware Faults. Do not modify it arbitrarily unless there are special requirements.

Chip Faults

Chip faults refer to basic software faults and chip hardware faults that occur on NPUs. With resumable training, processor faults are detected and reported by the device manager Ascend Device Plugin.

NPU Reporting Mechanism

When an NPU fault occurs, the fault management framework obtains the fault information and uploads it to the fault management framework of the NPU driver. After receiving the fault information, the fault management framework reports it to Ascend Device Plugin via the DCMI, as shown in Figure 1.

Ascend Device Plugin obtains chip health status through the DCMI. Currently, the following two acquisition modes are provided:

• Fault subscription mode: When Ascend Device Plugin starts, it first calls the DCMI fault subscription interface to register monitoring. When a fault occurs, the driver reports the fault event to Ascend Device Plugin through this interface. When the fault is recovered, the recovery event is reported to Ascend Device Plugin through this interface.
• Fault polling mode: At fixed intervals, the chip fault status is queried through the fault query interface. This mode is switched to when the device driver does not support the subscription capability.

Figure 1 Chip fault reporting

Ascend Device Plugin Reporting Mechanism

After obtaining chip fault information, Ascend Device Plugin reports it to K8s in the form of a ConfigMap. The fault reporting mechanism of Ascend Device Plugin is as follows:

Figure 2 Reporting faults to K8s

For different fault handling modes, the reporting paths differ.

• Rescheduling mode: After obtaining a chip fault, Ascend Device Plugin writes the chip fault information into device-info-cm of the node to which it belongs. For field descriptions, see the DeviceInfoCfg table. ClusterD reads device-info-cm of each node to detect chip faults and reports them to the scheduler.

  • Graceful fault tolerance mode: After obtaining a recoverable chip fault, Ascend Device Plugin writes the chip fault information into reset-info-cm of the job to which it belongs. The service container detects the chip fault by mounting reset-info-cm as a file and reading the file.

    If the graceful fault tolerance mode fails to handle the fault and falls back to the rescheduling mode, the fault reporting path follows the rescheduling mode.

Required Components

To ensure the proper functioning of the chip fault detection feature, the following components must be installed: Volcano, Ascend Operator, Ascend Device Plugin, and ClusterD.

(Optional) Configuring Fault Detection Levels

Resumable training provides the default fault frequency, duration, fault level, and fault handling policy for processor faults. If you want to modify the fault handling policy, see Chip Faults. Do not modify these settings unless you have specific requirements.

Supported Fault Handling Types

Job-level rescheduling, Pod-level rescheduling, process-level rescheduling, process-level online recovery, and graceful fault tolerance.

Parameter Plane Network Faults

NPU parameter plane network faults include chip network-related faults and UnifiedBus device faults.

When a fault occurs on the parameter plane network, it will cause training interruption or poor training task performance. After a fault occurs on the UnifiedBus device, MindCluster cluster scheduling components perform rescheduling based on the fault level.

Parameter plane network fault detection is handled by Ascend Device Plugin. Figure 1 shows the detailed workflow.

Figure 1 Fault detection

Key Steps

Chip network fault:

1. Each NPU periodically checks whether the communication with the gateway address is normal at an interval of 2.5 seconds and reports the result through the fault management framework.
2. The RoCE driver monitors the NPU network port link status in real time and reports Linkdown or Linkup events through the fault management framework.
3. Ascend Device Plugin obtains information from the fault management framework through the DCMI, queries the gateway detection results by polling, and subscribes to network port Linkdown or Linkup events in real time and reports them. Ascend Device Plugin counts the duration of abnormal gateway detection and the duration of Linkdown. If the duration is less than or equal to the RoCE network timeout (defaulted to 20 seconds), it is marked as an NPU network fault (not processed by default, which may cause a parameter plane network fault); if it exceeds 20 seconds, it is escalated to the configured fault level.

UnifiedBus device fault:

1. The UnifiedBus device writes the fault that occurred on the device into a local queue.
2. The UnifiedBus query interface queries the above queue, caches the fault to the query interface, and performs aggregation processing.
3. Ascend Device Plugin calls the interface through subscription or polling to obtain faults related to the UnifiedBus device, and writes them into device-info-cm for reporting.

Fault Reporting Mechanism

• When a chip network fault occurs, after the NPU fault management framework obtains the fault information, it reports the information to the NPU driver. After receiving the fault information, the NPU driver reports it to Ascend Device Plugin through the DCMI. Ascend Device Plugin then obtains the chip health status through the DCMI. Currently, the following two acquisition modes are provided:

  • Fault subscription mode: When Ascend Device Plugin starts, it first calls the DCMI fault subscription interface to register monitoring. When a fault occurs or is recovered, the driver reports the fault occurrence or recovery event to Ascend Device Plugin through this interface.
  • Fault polling mode: At fixed intervals, the chip fault status is queried through the fault query interface. This mode is switched to when the device driver does not support the subscription capability.

• When a fault occurs on the UnifiedBus device, Ascend Device Plugin obtains fault information through the UnifiedBus query interface. Currently, fault query provides two modes:

  • Fault subscription mode: During the startup of Ascend Device Plugin, a fault handling callback is registered with the UnifiedBus query interface. After a fault occurs, the callback is invoked to report the fault to Ascend Device Plugin, and when the fault is recovered, a recovery event is reported through this interface.
  • Fault polling mode: Ascend Device Plugin calls the full fault query interface once every 5 minutes.

Ascend Device Plugin Reporting Mechanism

After obtaining the parameter plane network fault, Ascend Device Plugin writes the fault information into device-info-cm and reports it to K8s in the form of a ConfigMap. For descriptions of each field in device-info-cm, see the DeviceInfoCfg table.

The fault reporting mechanism of Ascend Device Plugin is shown in Figure 2.

Figure 2 Fault reporting

Watchdog Fault Detection

A parameter plane network link anomaly (parameter plane network fault) may prevent normal NPUs in a job from communicating with the faulty NPU, causing all NPU collective communications to enter a timeout waiting state. The collective communication for the job exits only after a waiting timeout exception occurs, which is 30 minutes by default.

Enabling the watchdog function (and enabling the unconditional retry capability for service plane faults) can isolate the faulty NPU after a parameter plane network link anomaly occurs, reschedule the job to healthy NPUs, and thus enable the job to exit quickly within 6 minutes.

Required Components

To ensure the normal use of the parameter plane network fault detection function, the following components must be installed: Volcano, Ascend Operator, Ascend Device Plugin, and ClusterD.

Supported Fault Handling Types

Job-level rescheduling, Pod-level rescheduling, and process-level rescheduling.

(Optional) Configuring the Fault Detection Level

Resumable training provides the default fault level and fault handling policy for parameter plane faults. If you need to modify the fault handling policy, see Parameter Plane Network Faults. Do not modify it unless you have special requirements.

Service Plane Faults

Resumable training supports perceiving and handling job failures caused by service plane faults through Volcano (scheduler). A service plane fault occurs when all training processes within a container exit abnormally, causing the container to exit abnormally and the Pod status to change to Failed. In scenarios using Ascend Operator, service plane faults only support cases where some Pods of a job fail. If the status of all Pods in a job changes to Failed within a few seconds, the job will not be rescheduled and will be considered as failed.

Figure 1 shows the detection principle of service plane faults.

Figure 1 Detection principle

The scheduler continuously polls the Pod status of each job to perceive service plane faults and report them. You can handle service plane faults according to specific business requirements. After the resumable training feature detects a service plane fault, Volcano checks whether the unconditional retry function is enabled. If enabled, it reschedules the job to a new node that does not trigger rescheduling and re-executes it. Then the number of retry times decreases by 1. If the number of retry times is 0 or the unconditional retry function is disabled, the system does not handle the service container fault.

Watchdog Fault Detection

Abnormal job execution on an NPU (service plane fault) may prevent normal NPUs in the job from communicating with the faulty NPU, causing the collective communication of normal NPUs to enter a timeout waiting state. The job exits only after a collective communication wait timeout exception occurs (defaulted to 30 minutes). Enabling the watchdog function (which requires simultaneously enabling the unconditional retry capability for service plane faults) can isolate the faulty NPU after such an exception occurs and reschedule the job to a healthy NPU, thereby enabling the job to exit quickly within 6 minutes.

Required Components

To ensure the normal use of the service plane fault detection function, the following components must be installed: Volcano and Ascend Operator.

Supported Fault Handling Types

Job-level rescheduling, Pod-level rescheduling, process-level rescheduling, and graceful fault tolerance

Public Faults

Public faults refer to faults reported by other fault senders (non-MindCluster components), including NPU faults, node faults, network faults, and storage faults..

Reporting Mechanism

Upon fault detection, the public fault sender transmits the fault details to ClusterD through ConfigMap or gRPC. ClusterD summarizes the received information, writes it to cluster-info-device-cm, and reports it to Ascend-volcano-plugin.

• ConfigMapL: The fault discoverer writes fault information into a ConfigMap, and ClusterD obtains the fault information. You can call the ConfigMap interface to inject public faults by referring to ConfigMap.
• gRPC: The fault discoverer sends fault information to ClusterD through gRPC, and ClusterD obtains the fault information. You can call the gRPC interface to inject public faults by referring to gRPC Interface.

Figure 1 Public fault reporting

Required Components

To ensure proper functioning of the public fault detection feature, the following components must be installed.

• Mandatory components: Volcano, Ascend Operator, Ascend Device Plugin, ClusterD
• Optional component: NodeD

Supported Fault Handling Types

Job-level rescheduling, Pod-level rescheduling, and process-level rescheduling

(Optional) Configuring the Fault Detection Level and Sender

Resumable training provides the default fault level and supported fault sender for public faults. If you want to modify the fault level and fault sender of public faults, see Public Faults. Do not modify them unless you have special requirements.

Pingmesh UnifiedBus Network Faults

Refer to NPU network faults detected on the HCCS network within or across SuperPoDs.

Reporting Mechanism

NodeD calls the DCMI to start a pingmesh task and periodically queries the pingmesh results, writing the results to <nodename\>.log. By default, the file is stored in /user/mind-cluster/pingmesh both within the container and on the physical machine. However, the path on the physical machine can be changed as follows.

After obtaining the pingmesh results, ClusterD performs a preliminary analysis of the results and writes the fault information into a ConfigMap file named pingmesh-fault-<nodename>. ClusterD listens for information from this ConfigMap, aggregates the faults, and reports them to Volcano, which then performs scheduling.

Prerequisites

• (Required) A namespace has been created.
• The following components have been installed on the corresponding nodes: NodeD (required), Ascend Device Plugin (optional), ClusterD (optional).
• (Required) The NodeD startup parameter resultMaxAge has been configured.

Constraints

This feature is only supported by Atlas 900 A3 SuperPoD.

Configuring UnifiedBus Network Detection

To configure UnifiedBus network detection, perform the following steps.

1. Configure shared storage.

   ClusterD and NodeD interact through shared storage, and their shared storage root paths must be consistent. The owner of the shared directory root path is user 9000, which is the same as the user running ClusterD.

   1. Configure the server.

      

   2. Modify the NodeD configuration.

      

   3. If ClusterD exists, modify the ClusterD configuration.

      

   4. Run the kubectl get pods -o wide -A command. If the following example is displayed, the shared storage configuration is complete.

      

2. Enable or disable UnifiedBus network detection.

   • (Recommended) Ascend Device Plugin and ClusterD installed

     1. Log in to the environment and go to the NodeD decompression directory.

     2. Run the following command to create a ConfigMap file named pingmesh-config.

        pingmesh-config.yaml is the pingmesh configuration file, which can be obtained from the NodeD installation package.

        kubectl apply -f pingmesh-config.yaml

        The following is a response example:

        configmap/pingmesh-config created

     3. Run the following command to edit the pingmesh-config file. For instructions on filling in the parameters in this file, see Table 1.

        kubectl edit cm -n cluster-system pingmesh-config

        Table 1 pingmesh-config cm

        Parameter	Description	Value
        app	Key of one of the ConfigMap labels.	pingmesh
        global	Cluster configuration information.	-
        "1"	Configuration example for SuperPoD ID 1. Users can modify or add configurations based on actual conditions. When a SuperPoD is configured, NodeD uses the its configuration and ignores the global configuration.	SuperPoD ID
        activate	Whether to enable the pingmesh feature.	on or off
        task_interval	Pingmesh task interval, in seconds.	[1–60]

   • Ascend Device Plugin and ClusterD not installed.

     Create a ConfigMap named super-pod-<superPodID\> with the label app=pingmesh in the namespace cluster-system. The fields in this ConfigMap must be filled in according to the super-pod-<super-pod-id> table. An example is shown below.

     apiVersion: v1
     data:
       superPodDevice: '{"SuperPodID":"0","NodeDeviceMap":{"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"62914560","1":"62980097","10":"64225290","11":"64290827","12":"64487436","13":"64552973","14":"64749582","15":"64815119","2":"63176706","3":"63242243","4":"63438852","5":"63504389","6":"63700998","7":"63766535","8":"63963144","9":"64028681"}},"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"67108864","1":"67174401","10":"68419594","11":"68485131","12":"68681740","13":"68747277","14":"68943886","15":"69009423","2":"67371010","3":"67436547","4":"67633156","5":"67698693","6":"67895302","7":"67960839","8":"68157448","9":"68222985"}},"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"104857600","1":"104923137","10":"106168330","11":"106233867","12":"106430476","13":"106496013","14":"106692622","15":"106758159","2":"105119746","3":"105185283","4":"105381892","5":"105447429","6":"105644038","7":"105709575","8":"105906184","9":"105971721"}},"node-**-*":{"NodeName":"node-**-*","DeviceMap":{"0":"4194304","1":"4259841","10":"5505034","11":"5570571","12":"5767180","13":"5832717","14":"6029326","15":"6094863","2":"4456450","3":"4521987","4":"4718596","5":"4784133","6":"4980742","7":"5046279","8":"5242888","9":"5308425"}},"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"142606336","1":"142671873","10":"143917066","11":"143982603","12":"144179212","13":"144244749","14":"144441358","15":"144506895","2":"142868482","3":"142934019","4":"143130628","5":"143196165","6":"143392774","7":"143458311","8":"143654920","9":"143720457"}},"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"146800640","1":"146866177","10":"148111370","11":"148176907","12":"148373516","13":"148439053","14":"148635662","15":"148701199","2":"147062786","3":"147128323","4":"147324932","5":"147390469","6":"147587078","7":"147652615","8":"147849224","9":"147914761"}},"node-**-**":{"NodeName":"node-**-**","DeviceMap":{"0":"83886080","1":"83951617","10":"85196810","11":"85262347","12":"85458956","13":"85524493","14":"85721102","15":"85786639","2":"84148226","3":"84213763","4":"84410372","5":"84475909","6":"84672518","7":"84738055","8":"84934664","9":"85000201"}}}}'
     kind: ConfigMap
     metadata:
       labels:
         app: pingmesh
       name: super-pod-0       # 0 is the SuperPoD ID.
       namespace: cluster-system

Viewing Detection Results

The pingmesh results of UnifiedBus network detection are written to the file <nodename>.log. The detailed description of each field in this file is shown in the following table.

Table 2 <nodename>.log

Viewing Fault Information

Run the following command on the management node to view fault information detected for the UnifiedBus network.

kubectl describe cm -n cluster-system  pingmesh-fault-<nodename>

The table below describes each field in the fault information.

Table 3 pingmesh-fault-<nodename>

Known UnifiedBus Network Faults

Performance Degradation Faults

Using TaskD 7.1.RC1 or Later

MindCluster cluster scheduling components, together with the profiling capability provided by MindStudio, offers diagnostic functionality for performance degradation faults (slow nodes) within a cluster. This feature provides the capability of dynamic dotting and data persistence, allowing dotting to be enabled or disabled in real time without requiring job restart for diagnosis, ensuring uninterrupted training.

Table 1 shows supported data dotting types.

Table 1 Data dotting types

Constraints

• Currently, Step, SaveCheckpoint, FP, and DataLoader can only be enabled synchronously. To disable the above four types, Communication must be disabled at the same time.
• Communication operator data dotting can be enabled or disabled independently.
• Dynamic lightweight dotting and full dotting of MindStudio cannot be enabled at the same time. Enabling full dotting can cause data collection failures due to performance deterioration.

Prerequisites

• (Optional) ClusterD, Ascend Device Plugin, and Volcano have been installed (the versions of the above MindCluster components must be compatible with TaskD).
• Install torch_npu (optional; required for PyTorch scenarios; version ≥ 7.1.RC1), MindSpore (optional; required for MindSpore scenarios; version ≥ 2.7.0), CANN (mandatory; version ≥ 8.2.RC1), and TaskD (mandatory) in the container.

Preparing the Software Package

Table 2 Required packages

Configuring Performance Degradation Detection

This solution applies only to TaskD of version 7.1.RC1 or later. If you are using a component version earlier than 7.1.RC1, see the Using TaskD of Other Versions section.

• PyTorch

  1. Choose one of the following two methods as required.

     • Install mstx_torch_plugin in the container.

       1. Download mstx_torch_plugin.

       2. Install the package.

          pip install mstx_torch_plugin-1.0-py3-none-any.whl

       3. Import the .whl package in the AI task execution script.

          Ensure that it is imported after torch and torch_npu are imported.

          import torch
          import torch_npu
          import mstx_torch_plugin

     • If a non-native optimizer is used in the PyTorch scenario or mstx_torch_plugin is not used, you need to modify the training iteration in the training script by adding the step dotting code to obtain the time consumed by the training step.

       The following example is for the PyTorch-MindSpeed scenario. You need to modify the ./mindspeed_llm/training/training.py file and add the following bold fields.

         def train(forward_step_func, model, optimizer, opt_param_scheduler,
                   train_data_iterator, valid_data_iterator,
                   process_non_loss_data_func, config):
           # Cache into one-logger for callback
             ……
             ……
             if is_profile_enabled():
                 prof = get_profiler()
                 prof.start()
             step_id = iteration
             while iteration < args.train_iters:
                 stream = torch.npu.current_stream()      # Obtain the execution stream of the current environment to get the NPU-side time
                 range_id = torch.npu.mstx.range_start(f"step {step_id}", stream) # Mark the start of the current training step
                 ……
                 ……
                 if args.manual_gc:
                     if args.manual_gc_interval != 0 and iteration % args.manual_gc_interval == 0:
                         gc.collect()
       
                 if is_profile_enabled():
                     prof.step()
                 step_id +=1  # Increment the training step by one to identify the next step
                 torch.npu.mstx.range_end(range_id) # Mark the end of the current training step

  2. In the container, log in to the environment as the running user of the CANN package and run the source ${install_path}/set_env.sh command to set the environment variables. ${install_path} is the installation directory of the CANN software. An example is as follows.

     source /usr/local/Ascend/cann/set_env.sh

  3. Before starting training, import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload specified .so files. An example is as follows.

     export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so

     • libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. The default installation path is /usr/local/Ascend/cann/lib64/libmspti.so.

     • libtaskd.so: This .so file is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so.

       The TaskD installation path can be queried using the following command. The Location field in the response is the TaskD installation path.

       pip show taskd

  4. After the distributed environment is initialized and the global rank can be obtained, modify the training script to start TaskD Manager in the training script, start TaskD Proxy in the management process, and start TaskD Worker inside the training process.

     1. (Optional) Start TaskD Manager and TaskD Proxy. If lightweight profiling is enabled through the gRPC interface to obtain data written to disk, perform the following steps. If lightweight profiling is enabled through ConfigMap to obtain data written to disk, skip this step.

        1. Create the manager.py file and place it in the current directory when invoking the training script. The content of the manager.py file is as follows.

           from taskd.api import init_taskd_manager, start_taskd_manager
           import os
           
           job_id=os.getenv("MINDX_TASK_ID")
           node_nums=XX         # Total number of nodes
           proc_per_node=XX     # Number of training processes per node
           
           init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
           start_taskd_manager()

           For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

        2. Add the following code to the training script to start TaskD Manager and TaskD Proxy.

           sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py
           
           if [[ "${RANK}" -eq 0 ]]; then
               export MASTER_ADDR=${POD_IP}
               python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &      # The specific execution path of manager.py is determined by the current path, and the error.log path must be created in advance.
           fi
           
           torchrun ...

     2. Start TaskD Worker.

        The following example is for the PyTorch-MindSpeed scenario. You need to modify the QWEN3_for_PyTorch_2.7_code/mindspeed_llm/training/training.py file and add the following bold fields to the code.

         def pretrain(train_valid_test_dataset_provider,
                       model_provider,
                       model_type,
                       forward_step_func,
                       process_non_loss_data_func=None,
                       extra_args_provider=None,
                       args_defaults={}):
             print_rank_0('time to initialize megatron (seconds): {:.3f}'.format(
                 time.time() - _TRAIN_START_TIME))
             print_datetime('after megatron is initialized')
             import torch.distributed as dist
             if dist.is_initialized():
                 rank = dist.get_rank()
                 from taskd.api.taskd_worker_api import init_taskd_worker
                 from taskd.api.taskd_worker_api import start_taskd_worker
                 init_taskd_worker(rank,5000)
                 start_taskd_worker()
             app_metrics['app_model_init_finish_time'] = one_logger_utils.get_timestamp_in_ms()
             one_logger_utils.on_pretrain_start()

        In the code above, the input parameter 5000 in init_taskd_worker(rank,5000) is the upper limit size of /user/cluster-info/profiling. For details, see the upper_limit_of_disk_in_mb in def init_task_worker(rank_id: int, upper_limit_of_disk_in_mb: int = 5000, framework: str = "pt") -> bool.

  5. Modify the job YAML.

     1. Modify the container port and add port 9601 for TaskD communication under all Pods.

        ...
              spec:
        ...
                containers:
        ...
                  ports:
                  - containerPort: 9601
                    name: taskd-port
        ...

     2. Mount files.

        1. Mount the lightweight profiling configuration file: The data-trace ConfigMap corresponding to the job on the host must be persisted to the /user/cluster-info/datatrace-config/Namespace.data-trace-Job name/ directory. Mount the file named profilingSwitch to the specified container path /user/cluster-info/datatrace-config/.
        2. Mount the lightweight profiling persistent file: Lightweight profiling data is written to the /user/cluster-info/profiling path in the container. To obtain it on the host, modify the job YAML to mount this path externally.

           • The following is an example of YAML mounting inside a container.

             volumeMounts:
             - name: profilingdata
               mountPath: /user/cluster-info/
             - name: profileswitch
               mountPath: /user/cluster-info/datatrace-config

           • The following is an example of YAML mounting on the host.

             volumes:
             - name: profileswitch
               hostPath:
                 path: /user/cluster-info/datatrace-config/default.data-trace-default-test-pytorch-fault-mixtral
             - name: profilingdata
               hostPath:
                 path: /home/profilingdatapath

  6. Enable lightweight profiling to obtain data written to disk. The following two methods are supported:

     • Modify the gRPC interface provided by ClusterD: If 4.a is configured, you need to use this method to enable it. For detailed interface information, see ModifyTrainingDataTraceSwitch.

       When you enable or modify lightweight profiling to obtain data written to disk through the gRPC interface provided by ClusterD, the lifecycle of the created data-trace-<\Job name> ConfigMap is deleted along with the job. If the job does not exist, the interface fails to be called.

     • Modify the data-trace ConfigMap corresponding to the job. If 4.a is not configured, you need to use this method to enable it. The specific steps are as follows:

       Taking the job named default-test-pytorch-fault-mixtral in the default namespace as an example, enable lightweight profiling to obtain data written to disk by editing the ConfigMap. An example is shown below.

       1. Run the following command on the master node to query the configuration ConfigMap.

          kubectl get cm

          • If data-trace-default-test-pytorch-fault-mixtral cm already exists, perform Step 3 to edit the file.

            The response example is as follows:

            NAME                                              DATA   AGE
            data-trace-default-test-pytorch-fault-mixtral     1      18h

          • If data-trace-default-test-pytorch-fault-mixtral cm does not exist, perform Step 2 to create the file.

       2. Run the following command to create the ConfigMap file required for configuring lightweight profiling to obtain data written to disk.

          1. Write the following content into datacm.yaml.

             apiVersion: v1
             kind: ConfigMap
             metadata:
               name: data-trace-default-test-pytorch-fault-mixtral  # The cm name must use the prefix data-trace + the job name.
               labels:
                 reset: "true"
             data:
               profilingSwitch: '{"CommunicationOperator":"off","Step":"on","SaveCheckpoint":"on","FP":"on","DataLoader":"on"}'

          2. Run the following command on the master node to create the ConfigMap.

             kubectl apply -f datacm.yaml

             The response is displayed as follows, indicating that the ConfigMap is created successfully.

             configmap/data-trace-default-test-pytorch-fault-mixtral created

       3. Run the following command to edit the ConfigMap file.

          kubectl edit cm data-trace-default-test-pytorch-fault-mixtral

       4. To enable communication operators, change the value of the CommunicationOperator field to on.

          apiVersion: v1
          data:
            profilingSwitch: '{"CommunicationOperator":"on","Step":"on","SaveCheckpoint":"on","FP":"on","DataLoader":"on"}'

          Enabling communication operators may deteriorate training performance. Therefore, you are advised not to enable them.

       5. Press Esc, enter :wq! to save and exit.

• MindSpore

  1. Inside the container, log in as the operating user of the CANN package and run the source ${install_path}/set_env.sh command to set environment variables. ${install_path} is the installation directory of the CANN software. An example is as follows.

     source /usr/local/Ascend/cann/set_env.sh

  2. Before starting training, import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload the specified .so file. An example is as follows.

     export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so

     • libmspti.so: This .so file is provided by MindStudio and integrated in the CANN package. If the default installation path is used, the path is /usr/local/Ascend/cann/lib64/libmspti.so.

     • libtaskd.so: This .so file is provided by TaskD. After installing the whl package, the path is TaskD path/taskd/python/cython_api/libs/libtaskd.so.

       The path where TaskD is located can be queried using the following command. The Location field in the response is the path where TaskD is located.

       pip show taskd

  3. After the distributed environment initialization is complete and the global rank can be obtained, modify the training script to start TaskD Manager in the training script, start TaskD Proxy in the management process, and start TaskD Worker in the training process.

     1. (Optional) Start TaskD Manager and TaskD Proxy. If lightweight profiling is enabled through the gRPC interface to obtain data written to disk, perform the following steps; if lightweight profiling is enabled through ConfigMap to obtain data written to disk, skip this step.

        1. Create a manager.py file in the current directory when calling the training script. The content of the manager.py file is as follows.

           from taskd.api import init_taskd_manager, start_taskd_manager
           import os
           
           job_id=os.getenv("MINDX_TASK_ID")
           node_nums=XX         # Total number of nodes
           proc_per_node=XX     # Number of training processes per node
           
           init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
           start_taskd_manager()

           For detailed parameter descriptions in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.

        2. Add the following code to the training script to start TaskD Manager.

           if [[ "${MS_SCHED_HOST}" -eq "${POD_IP}" ]]; then
               python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &       # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
           fi
           
           msrun ...

        3. Modify the mindspore/python/mindspore/parallel/cluster/process_entity/_api.py file to start TaskD Proxy. An example is shown below.

            ...
              if ("TTP:1" in tft_env) or ("UCE:1" in tft_env) or ("ARF:1" in tft_env):
                        try:
                            from taskd.python.framework.agent.ms_mgr.msrun_plugin import MSRunPlugin
                            from taskd.api.taskd_proxy_api import init_taskd_proxy
                            from taskd.python.framework.common.type import CONFIG_UPSTREAMIP_KEY, LOCAL_HOST
                            import threading
                            proxy = threading.Thread(target=init_taskd_proxy, args=({CONFIG_UPSTREAMIP_KEY : os.getenv("MS_SCHED_HOST", LOCAL_HOST)},))
                            proxy.daemon = True
                            proxy.start()
                            self.msmgr = MSRunPlugin()
                            self.msmgr.register_callbacks("KILL_WORKER", self.kill_workers)
                            self.msmgr.register_callbacks("START_ALL_WORKER", self.start_all_workers)
                            self.msmgr.register_callbacks("START_WORKER_LIST", self.start_worker_list)
                            self.msmgr.register_callbacks("MONITOR", self.monitor_rank_status)
                            self.enable_mindx = True
                            os.environ["MS_ENABLE_RECOVERY"] = str(1)
            ...

     2. Start TaskD Worker.

        The following example is for the MindSpore-MindFormers scenario. You need to modify the ./mindformers/trainer/base_trainer.py file and add the following bold fields to the code.

             def training_process(
                     self,
                     config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None,
                     network: Optional[Union[Cell, PreTrainedModel]] = None,
                     dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None,
                     optimizer: Optional[Optimizer] = None,
                     callbacks: Optional[Union[Callback, List[Callback]]] = None,
                     compute_metrics: Optional[Union[dict, set]] = None,
                     **kwargs):
                 ……
                 ……
        
                 logger.info(".........Starting Training Model..........")
                 if get_real_rank() % 8 == 0:
                     pprint(config)
                 logger.info(".........Model Compiling, Please Wait a Moment...........")
                 try:
                     rank = get_rank()
                     from taskd.api.taskd_worker_api import init_taskd_worker
                     from taskd.api.taskd_worker_api import start_taskd_worker
                     init_taskd_worker(rank,5000)
                     start_taskd_worker()
                 except Exception as e:
                     print("failed to call mindcluster taskd")
                 model.train(config.runner_config.epochs, dataset,
                             callbacks=callbacks,
                             dataset_sink_mode=config.runner_config.sink_mode,
                             sink_size=config.runner_config.sink_size,
                             initial_epoch=config.runner_config.initial_epoch)

        The input parameter 5000 in the above code init_taskd_worker(rank,5000) is the upper limit size of /user/cluster-info/profiling. For details, see the upper_limit_of_disk_in_mb parameter in def init_taskd_worker(rank_id: int, upper_limit_of_disk_in_mb: int = 5000, framework: str = "pt") -> bool.

  4. Modify the job YAML. For details, see Step 5 in the PyTorch scenario.

  5. Enable lightweight profiling to obtain data written to disk. For details, see Step 6 in the PyTorch scenario.

Obtaining Performance Degradation Detection Data

• Data written to disk is classified by rank. Lightweight profiling data is written to the /user/cluster-info/profiling path in the container.

• For Pods with the environment variable MINDX_TAS_ID, the path of rank 0 is /user/cluster-info/profiling/$MINDX_TASK_ID/0.

  • If this environment variable is not present, data is written by default to a folder named default_task_id_timestamp.
  • When /user/cluster-info/profiling reaches the configured maximum size (refer to 4.b for PyTorch scenarios; refer to 3.b for MindSpore scenarios), file aging is triggered. By default, the oldest 20% of files are deleted each time. During the aging process, only numerically named files within the rank folders under the profiling directory are deleted. It is recommended not to manually add other files to the profiling folder. If users manually add other files, TaskD will not delete them, but these files will occupy space.
  • Lightweight profiling files are named with timestamps, each record is separated by a newline, and data is appended to the latest file under the rank directory each time. When the latest file exceeds 10 MB, TaskD creates a new profiling file. If network storage methods such as NFS are used, a new file may be created before the file size reaches 10 MB due to slow data synchronization.

Using TaskD of Other Versions

MindCluster cluster scheduling components provide the diagnosis function for performance degradation (slow nodes) in a cluster based on the profiling capability provided by MindStudio. This function provides the capability of dynamic dotting and data persistence, allowing dotting to be enabled or disabled in real time without requiring job restart for diagnosis, ensuring uninterrupted training.

Table 1 describes the supported dotting data.

Table 1 Dotting data description

Constraints

• Currently, Step, SaveCheckpoint, FP, and DataLoader can only be enabled synchronously. To disable the above four data point types, Communication must also be disabled at the same time.
• Communication operator data dotting can be enabled or disabled separately.
• Dynamic lightweight dotting and full dotting of MindStudio cannot be enabled at the same time. Enabling full dotting can cause data collection failures due to performance deterioration.

Prerequisites

• (Optional) ClusterD, Ascend Device Plugin, and Volcano have been installed (the versions of the above MindCluster components must be compatible with TaskD).
• Install torch_npu (optional; required for PyTorch scenarios; version ≥ 7.0.0), MindSpore (optional; required for MindSpore scenarios; version ≥ 2.6.RC1), CANN (mandatory; version ≥ 8.1.RC1), and TaskD (mandatory, version ≥ 7.0.RC1) in the container.

Preparing the Software Package

Table 2 Preparing software packages

Configuring Performance Degradation Detection

This solution applies only to TaskD of versions earlier than 7.1.RC1. If you are version 7.1.RC1 or later, see the Using TaskD 7.1.RC1 or Later section.

• PyTorc

  1. (Optional) Install mstx_torch_plugin in the container.

     1. Download mstx_torc_plugin.

     2. Install the package.

        pip install mstx_torch_plugin-1.0-py3-none-any.whl

     3. Import the whl package in the AI task execution script.

        Ensure that it is imported after torch and torch_npu are imported.

        import torch
        import torch_npu
        import mstx_torch_plugin

  2. (Optional) If a non-native optimizer is used in the PyTorch scenario or mstx_torch_plugin is not used, you need to modify the training iteration in the training script by adding the step dotting code to obtain the time consumed by the training step.

     The following example is for the PyTorch-MindSpeed scenario. You need to modify the ./mindspeed_llm/training/training.py file and add the following bold fields.

      def train(forward_step_func, model, optimizer, opt_param_scheduler,
                train_data_iterator, valid_data_iterator,
                process_non_loss_data_func, config):
        # Cache into one-logger for callback
          ……
          ……
          if is_profile_enabled():
              prof = get_profiler()
              prof.start()
          step_id = iteration
          while iteration < args.train_iters:
             stream = torch.npu.current_stream()      # Obtains the execution stream of the current environment, used to get the NPU-side time
              range_id = torch.npu.mstx.range_start(f"step {step_id}", stream) # Marks the start of the current training step
              ……
              ……
              if args.manual_gc:
                  if args.manual_gc_interval != 0 and iteration % args.manual_gc_interval == 0:
                      gc.collect()
     
              if is_profile_enabled():
                  prof.step()
              step_id +=1  # Increments the training step by one, used to identify the next step
              torch.npu.mstx.range_end(range_id) # Marks the end of the current training step

  3. In the container, log in to the environment as the running user of the CANN package and run the source ${install_path}/set_env.sh command to set environment variables. Here, ${install_path} is the installation directory of the CANN software. An example is shown below.

     source /usr/local/Ascend/cann/set_env.sh

  4. Before starting training, import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload specified .so files. An example is shown below.

     export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so

     • libmspti.so: This so is provided by MindStudio and integrated in the CANN package. If the default installation path is used, the path is /usr/local/Ascend/cann/lib64/libmspti.so.

     • libtaskd.so: This so is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so.

       The TaskD installation path can be queried using the following command. The Location field in the response is the TaskD installation path.

       pip show taskd

  5. After the distributed environment is initialized and the global rank is obtained, modify the training script to start TaskD Worker inside the training process.

     The following example is for the PyTorch-MindSpeed scenario. You need to modify the QWEN3_for_PyTorch_2.7_code/mindspeed_llm/training/training.py file and add the following bold fields in the code.

        def pretrain(train_valid_test_dataset_provider,
                      model_provider,
                      model_type,
                      forward_step_func,
                      process_non_loss_data_func=None,
                      extra_args_provider=None,
                      args_defaults={}):
            print_rank_0('time to initialize megatron (seconds): {:.3f}'.format(
                time.time() - _TRAIN_START_TIME))
            print_datetime('after megatron is initialized')
            import torch.distributed as dist
            if dist.is_initialized():
                rank = dist.get_rank()
                from taskd.api.taskd_worker_api import init_taskd_worker
                from taskd.api.taskd_worker_api import start_taskd_worker
                init_taskd_worker(rank,5000)
                start_taskd_worker()
            app_metrics['app_model_init_finish_time'] = one_logger_utils.get_timestamp_in_ms()
            one_logger_utils.on_pretrain_start()

     The input parameter 5000 in the above code init_taskd_worker(rank,5000) is the upper limit size of /user/cluster-info/profiling. For detailed description, see the "upper_limit_of_disk_in_mb" parameter in def init_taskd_worker(rank_id: int, upper_limit_of_disk_in_mb: int = 5000, framework: str = "pt") -> bool.

  6. Modify the job YAML.

     1. MMount the lightweight profiling configuration file: You need to flush the data-trace ConfigMap on the host to the /user/cluster-info/datatrace-config/Namespace.data-trace-Job name/ folder. Mount the profilingSwitch file to the specified path /user/cluster-info/datatrace-config/ in the container.
     2. Mount the lightweight profiling disk file: Lightweight profiling data is written to the /user/cluster-info/profiling path inside the container. To obtain it on the host, modify the job YAML to mount this path out.

        • The following is an example of YAML mounting inside a container.

          volumeMounts:
          - name: profilingdata
            mountPath: /user/cluster-info/
          - name: profileswitch
            mountPath: /user/cluster-info/datatrace-config

        • The following is an example of YAML mounting on the host.

          volumes:
          - name: profileswitch
            hostPath:
              path: /user/cluster-info/datatrace-config/default.data-trace-default-test-pytorch-fault-mixtral
          - name: profilingdata
            hostPath:
              path: /home/profilingdatapath

  7. Enable lightweight profiling to obtain data written to disk. Modify the data-trace ConfigMap corresponding to the task or the gRPC interface provided by ClusterD (see ModifyTrainingDataTraceSwitch for interface details) to dynamically enable or disable the lightweight profiling capability.

     The following example uses the job named default-test-pytorch-fault-mixtral in the default namespace and enables lightweight profiling to obtain data written to disk by editing the ConfigMap.

     1. Run the following command on the master node to query the ConfigMap corresponding to the job.

        kubectl get cm

        • If data-trace-default-test-pytorch-fault-mixtral cm already exists, perform Step 3 to edit the file.

          The response example is as follows:

          NAME                                              DATA   AGE
          data-trace-default-test-pytorch-fault-mixtral     1      18h

        • If data-trace-default-test-pytorch-fault-mixtral cm does not exist, perform Step 2 to create the file.

     2. Run the following command to create the ConfigMap file required for configuring lightweight profiling to obtain data written to disk.

        1. Write the following content into datacm.yaml.

           apiVersion: v1
           kind: ConfigMap
           metadata:
             name: data-trace-default-test-pytorch-fault-mixtral  # The ConfigMap name must start with the prefix `data-trace` followed by the job name.
             labels:
               reset: "true"
           data:
             profilingSwitch: '{"CommunicationOperator":"off","Step":"on","SaveCheckpoint":"on","FP":"on","DataLoader":"on"}'

        2. Run the following command on the master node to create the ConfigMap.

           kubectl apply -f datacm.yaml

           The following response indicates that the ConfigMap is created successfully.

           configmap/data-trace-default-test-pytorch-fault-mixtral created

     3. Run the following command to edit the ConfigMap file.

        kubectl edit cm data-trace-default-test-pytorch-fault-mixtral

     4. To enable communication operators, change the value of the CommunicationOperator field to on.

        apiVersion: v1
        data:
          profilingSwitch: '{"CommunicationOperator":"on","Step":"on","SaveCheckpoint":"on","FP":"on","DataLoader":"on"}'

        Enabling communication operators may degrade training performance. It is not recommended to keep communication operators enabled in normal conditions.

     5. Press Esc, enter :wq! to save and exit.

• MindSpore

1. In the container, log in to the environment as the running user of the CANN package and run the source ${install_path}/set_env.sh command to set environment variables. ${install_path} is the installation directory of the CANN software. Example:

   source /usr/local/Ascend/cann/set_env.sh

2. Before starting training, import the LD_PRELOAD environment variable in the training script. This environment variable allows the system to preload specified .so files. Example:

   export LD_PRELOAD=/usr/local/Ascend/cann/lib64/libmspti.so:/usr/local/python3.10.5/lib/python3.10/site-packages/taskd/python/cython_api/libs/libtaskd.so

   • libmspti.so: This .so file is provided by MindStudio and integrated into the CANN package. If the default installation path is used, the path is /usr/local/Ascend/cann/lib64/libmspti.so.

   • libtaskd.so: This .so file is provided by TaskD. After the whl package is installed, the path is TaskD installation path/taskd/python/cython_api/libs/libtaskd.so.

     The TaskD installation path can be queried using the following command. The Location field in the response is the TaskD installation path.

     pip show taskd

3. After the distributed environment initialization is complete and the global rank can be obtained, modify the training script to start TaskD Worker inside the training process.

   The following example is for the MindSpore-MindFormers scenario. You need to modify the ./mindformers/trainer/base_trainer.py file and add the following bold fields to the code.

    <pre codetype="Python">
        def training_process(
                self,
                config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None,
                network: Optional[Union[Cell, PreTrainedModel]] = None,
                dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None,
                optimizer: Optional[Optimizer] = None,
                callbacks: Optional[Union[Callback, List[Callback]]] = None,
                compute_metrics: Optional[Union[dict, set]] = None,
                **kwargs):
            ……
            ……
   
            logger.info(".........Starting Training Model..........")
            if get_real_rank() % 8 == 0:
                pprint(config)
            logger.info(".........Model Compiling, Please Wait a Moment...........")
            <strong>try:</strong>
                <strong>rank = get_rank()</strong>
                <strong>from taskd.api.taskd_worker_api import init_taskd_worker</strong>
                <strong>from taskd.api.taskd_worker_api import start_taskd_worker</strong>
                <strong>init_taskd_worker(rank,5000)</strong>
                <strong>start_taskd_worker()</strong>
            <strong>except Exception as e:</strong>
                <strong>print("failed to call mindcluster taskd")</strong>
            model.train(config.runner_config.epochs, dataset,
                        callbacks=callbacks,
                        dataset_sink_mode=config.runner_config.sink_mode,
                        sink_size=config.runner_config.sink_size,
                        initial_epoch=config.runner_config.initial_epoch)</pre>

   In the code above, the input parameter 5000 in init_taskd_worker(rank,5000) is the upper limit size of /user/cluster-info/profiling. For details, see the "upper_limit_of_disk_in_mb parameter in def init_taskd_worker(rank_id: int, upper_limit_of_disk_in_mb: int = 5000, framework: str = "pt") -> bool.

4. Modify the job YAML. For details, see Step 6 in the PyTorch Scenario.

5. Enable lightweight profiling to obtain data flushed to disk. For details, see Step 7 in the PyTorch Scenario.

Obtaining Performance Degradation Detection Data

• The data written to disk is classified by rank. Lightweight profiling data is written to the /user/cluster-info/profiling path inside the container.

• For Pods with the environment variable MINDX_TASK_ID, the rank 0 data path inside the container is /user/cluster-info/profiling/$MINDX_TASK_ID/0.

  • If this environment variable does not exist, data is written to a folder named default_task_id_timestamp</i> by default.
  • When/user/cluster-info/profiling reaches the configured upper limit (for PyTorch scenarios, refer to Step 5; for MindSpore scenarios, refer to Step 3), file aging is triggered. By default, the oldest 20% of files are deleted each time. During the aging process, only numerically named files in the rank folders under the profiling directory are deleted. It is recommended not to manually add other files to the profiling folder. If users manually add other files, TaskD will not delete them, but these files will occupy space.
  • Lightweight profiling files are named with timestamps. Each record is separated by a newline and appended to the latest file under the rank. When the latest file exceeds 10 MB, TaskD creates a new profiling file. If network storage methods such as NFS are used, a new file may be created before the file size reaches 10 MB due to slow data synchronization.

Slow Nodes & Slow Network Faults

Introduction

MindCluster cluster scheduling components, together with MindCluster Ascend FaultDiag (fault diagnosis tool), provide the diagnostic function for slow nodes and slow network faults in a cluster.

Prerequisites

Before using the slow node & slow network fault diagnostic function, you need to increase the CPU and memory resource sizes in NodeD and change the resource information in the NodeD startup YAML file.

The current YAML file content is as follows:

resources:
            requests:
              memory: 300Mi
              cpu: 500m
            limits:
              memory: 300Mi
              cpu: 500m

The modified YAML file content is as follows:

resources:
            requests:
              memory: 10Gi
              cpu: 5000m
            limits:
              memory: 10Gi
              cpu: 5000m

Deployment Mode

ClusterD and the Fault Diagnose Online (FD-OL) framework are deployed in one process on the management node. Once ClusterD is started, FD-OL is automatically started..

Slow Node Diagnosis

Function Description

For performance degradation of node training in AI clusters, it supports real-time detection of slow nodes caused by computing domain issues or network problems, allowing users to isolate slow nodes via switchover or other methods.

Currently, only online deployment integrated with ClusterD and NodeD is supported. See the Installation and Deployment chapter to complete the deployment of ClusterD and NodeD.

• Slow node algorithm: Based on key performance indicators of training scenarios, it perceives real-time degradation status. For the synchronization relationship between communication operators and computing operators, it achieves problem demarcation for slow computing cards and slow communication domains.
• Slow node cleaning: Converts and cleans incremental data within nodes, generating a cleaning result CSV file.
• Slow node scheduling: Schedules the overall process of slow nodes and controls data cleaning and the slow node algorithm.

Prerequisites

The deployment of performance degradation faults has been completed.

Usage Example

Procedures for starting a diagnosis task on slow nodes:

1. Add a function call to obtain parallel domain information in the training iteration of the training script. The following uses the PyTorch-MindSpeed scenario as an example. You need to add the following fields in bold to the ./mindspeed_llm/training/training.py file.

    def train(forward_step_func, model, optimizer, opt_param_scheduler,
              train_data_iterator, valid_data_iterator,
              process_non_loss_data_func, config):
        ……
        if is_profile_enabled():
            prof = get_profiler()
            prof.start()
        m_iter = 0
        while iteration < args.train_iters:
            ……
            args.curr_iteration = iteration
            loss_dict, skipped_iter, grad_norm, num_zeros_in_grad = \
                train_step(forward_step_func,
                           train_data_iterator,
                           model,
                           optimizer,
                           opt_param_scheduler,
                           config)
            iteration += 1
            m_iter += 1
            if m_iter == 5:
                from taskd.python.adaptor.pytorch.group_info import dump_group_info
                dump_group_info()
            batch_size = mpu.get_data_parallel_world_size() * \
                         args.micro_batch_size * \
                         get_num_microbatches()

2. Complete operations in preparations before use and deployment form.

3. Run the kubectl apply -f ajob-2pod-16npu.yaml command to create a slow node diagnosis task and write it to the configMap.

   

4. The content of ajob-2pod-16npu.yaml is as follows. For details about the command output, see Table 1.

   

   The following is a YAML example, which cannot be directly copied, compiled, or run. It is for reference only.

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ras-feature-slownode-default-test-pytorch-2pod-16npu    # The value of JobName must be the same as the name attribute of the following job. The prefix ras-feature-slownode- cannot be modified.
     namespace: mindx-dl
     labels:
       fd-ol-slow-node: "true"
   data:
     FeatConf: |
       {"jobName":"default-test-pytorch-2pod-16npu","jobNamespace":"default","normalNumber":20,"nSigma":3,"degradationPercentage":0.3,"nConsecAnomaliesSignifySlow":3,"nSecondsDoOneDetection":30,"clusterMeanDistance":1.3,"cardOneNode":16,"SlowNode":1}
   ---

   Table 1 YAML file description

   Field	Default Value	Description
   jobNamespace	default	Job namespace.
   jobName	-	Job name
   normalNumber	20	Initial computing threshold (normal quantity).
   nSigma	3	Number of sigmas for calculating upper and lower thresholds
   degradationPercentage	0.3	Deterioration rate. A value of 0.3 represents a 30% performance drop.
   nConsecAnomaliesSignifySlow	3	Number of exceptions. Detection is triggered only when exceptions occur for multiple consecutive times.
   nSecondsDoOneDetection	30s	Interval for detection, in seconds
   clusterMeanDistance	1.3	Threshold distance (mean1 and mean2) between two clusters after clustering
   cardOneNode	16	Number of cards on a node
   slowNode	1	Whether to enable the job. 1: enabled 0: disabled

Querying Slow Node Diagnosis Results

After creating a slow node diagnosis task, you can query the logs of ClusterD and NodeD to view the task details.

Method 1: Querying Cluster-Side Slow Node Diagnosis Logs via K8s Logs

1. Run the kubectl get pods -n mindx-dl command to query the data of started ClusterD and NodeD nodes.

   

2. Then, run the kubectl logs -n mindx-dl clusterd-7d5db546d8-kdslz | grep "got degradation, slow rank" command to query the log data.

3. Check the log information. If information similar to the following is displayed, the node deteriorates.

   

Method 2: Querying Cluster-Side Slow Node Diagnosis Logs via Flushed Logs

1. Run the cat /var/log/mindx-dl.clusterd.clusterd.log | grep "got degradation, slow rank" command to query log data.

2. Check the log information. If information similar to the following is displayed, the node deteriorates.

   

Method 3: Querying Slow Node Diagnosis Logs on a Node

Run the kubectl logs -n mindx-dl node-9ld8k | grep "is degradation" command to query the log data. If the information similar to the following is displayed, the node deteriorates.

Known Slow Node Faults

Slow Network Diagnosis

Function Description

This feature provides parameter plane connectivity checks, real-time monitoring, and proactive risk warnings. By streamlining fault diagnostics and demarcation, it pre-warns network issues and sub-healthy faults and ensures the long-term stability of the cluster network.

Currently, it only supports online deployment integrated with ClusterD and NodeD. See the Installation and Deployment section to complete the deployment of ClusterD and NodeD.

• Slow network algorithm: Analyzes and detects network probing data between nodes, and outputs network diagnosis results.
• Slow network scheduling: Controls the start and stop of detection tasks, reports fault results, and schedules the overall slow network process.

Usage Example

1. Configure shared storage.

   ClusterD and NodeD interact through shared storage, and their shared storage root paths must be consistent. The owner of the shared directory root path is user 9000, which is the same as the user running ClusterD.

   1. Configure the server.

      

   2. Modify the NodeD configuration.

      

   3. Modify the ClusterD configuration.

      

   4. Run the kubectl get pods -o wide -A command. If the following example output appears, the shared storage configuration is complete.

      

2. Enable the fault detection switch.

   1. Log in to the environment and go to the NodeD decompression directory.

   2. Run the following command to create a ConfigMap file named pingmesh-config. pingmesh-config.yaml is the pingmesh configuration file, which can be obtained from the NodeD installation package.

      kubectl apply -f pingmesh-config.yaml

      The response example is as follows:

      configmap/pingmesh-config created

   3. Run the following command to edit the pingmesh-config file. The description of each parameter in this file is shown in the following table.

      kubectl edit cm -n cluster-system pingmesh-config

      Table 1 pingmesh-config file parameter description

      Parameter	Value	Description
      app	pingmesh	Key of a label in the ConfigMap
      global	-	Cluster configuration
      "1"	SuperPoD ID	Configuration example for SuperPoD ID 1. Modify or add configurations based on actual conditions. When a SuperPoD is configured, NodeD uses the configuration of the SuperPoD and ignores the global configuration.
      activate	on: enabled off: disabled	Whether to enable the pingmesh function.
      task_interval	[1–60]	Interval for executing a pingmesh task, in seconds.

Viewing Detection Results

The pingmesh results of network detection are written to the file <nodename>.log. The detailed description of each field in this file is shown in the following table.

Table 2 Parameter description of the <nodename>.log file

Viewing gRPC Report Results

If a slow network fault is detected, the fault is reported to the public fault management center of ClusterD through gRPC.

If a slow network fault is detected, the fault is reported to the public fault management center of ClusterD through gRPC.

Known Slow Network Faults

Fault Handling

Fault Decision Description

Once fault detection is complete, resumable training can restore the training service through fault handling or tolerance mechanisms across different fault modes, including Job-level rescheduling, Pod-level rescheduling, process-level rescheduling, elastic training, operator-level online recovery, and process-level online recovery. You can choose the appropriate sub-feature based on your requirements.

Figure 1 Fault handling description

In the figure above, Mean Time to Repair (MTTR) represents the duration from fault occurrence to recovery. Success rate measures the effectiveness of fault recovery after an issue arises. Usability evaluates the cost of implementing or integrating a fault policy.

Job-level rescheduling, pod-level rescheduling, and process-level rescheduling support all fault modes supported by resumable training, but depend on backup redundant compute server resources. If there is an unrecoverable hardware fault and no backup redundant compute server, you can configure elastic training to perform scale-in training. Process-level online recovery is applicable to on-chip memory faults and network faults. Operator-level online recovery supports processor network faults and UnifiedBus network faults.

The multi-layer fault handling system of resumable training supports rollback at each layer based on recovery granularity, as shown in Figure 2. If recovery at a higher layer fails, the process can revert to the next lower layer.

Figure 2 Recovery failure description

Rescheduling Mode

1. Rescheduling mode: Schedules jobs to healthy chips and isolates faulty chips.

   The default rescheduling mode is Job-level rescheduling, which stops all Pods upon each fault. However, for large-scale jobs, the cost of stopping all Pods before rescheduling is high, and the fault recovery time is excessively long. In addition, resumable training also provides the Pod-level rescheduling function. You can configure it based on the job scale, so that only the Pods related to the fault are stopped and a small number of Pods are rescheduled upon a fault, thereby achieving rapid fault recovery. To further shorten the fault recovery time and reduce the fault impact scope, resumable training also provides process-level rescheduling and process-level online recovery functions.

   Table 1 Differences between various rescheduling levels

   Rescheduling Level	Recovery Time	Configuration Procedure	Description
   Job-level rescheduling	Job-level rescheduling has a long recovery time, which degrades superlinearly as the job scale increases.	The operation steps for Job-level rescheduling are simple. Users of MindCluster only need to enable the configuration switch to use it. For key configuration procedures, see Configuring Job-Level Rescheduling.	To further reduce the resource scheduling time during recovery, you can choose to enable Pod-level rescheduling on top of Job-level rescheduling.
   Pod-level rescheduling	Pod-level rescheduling can shorten the resource scheduling time and is independent of the job scale. However, Pod-level rescheduling cannot optimize the time overhead during training initialization, and the overall recovery time still degrades superlinearly as the job scale increases.	For Pod-level rescheduling, you need to additionally integrate training process management capabilities into the training container. Users of MindCluster can use it after acquiring the corresponding process management capabilities. For key configuration procedures, see Configuring Pod-Level Rescheduling.	To further reduce the recovery time during training initialization, you can choose to enable process-level rescheduling on top of Pod-level rescheduling.
   Process-level rescheduling (process-level recovery)	Process-level rescheduling can reduce the training initialization time, shorten the overall recovery time, and is independent of or weakly correlated with the job scale.	Compared with Pod-level rescheduling, process-level rescheduling requires you to additionally integrate high-availability training capabilities into the training framework. Users of MindCluster need to modify the training script and enable the corresponding configuration switch to use it. For key configuration procedures, see Configuring Process-Level Rescheduling.	To address the issue of short MTBF in large-scale scenarios and further reduce the overall recovery time, you can choose to enable process-level online recovery on top of process-level rescheduling.
   Process-level online recovery	Process-level online recovery has a lower recovery time compared with process-level rescheduling.	Compared with process-level rescheduling, process-level online recovery requires users to configure the corresponding configuration switch before use. For key configuration procedures, see Configuring Process-Level Online Recovery.	Currently, process-level online recovery supports on-chip memory faults and network faults. Other fault scenarios will fall back to other handling methods.
   Operator-level online recovery	-	For key configuration procedures, see Configuring Operator-Level Online Recovery.	-

2. The rescheduling mode has the following two rescheduling policies.

   • Direct rescheduling: If a hardware fault that can be detected by the cluster scheduling components occurs during training, the system isolates the faulty node or processor and directly reschedules the job.
   • Unconditional retry: If a fault that cannot be detected by the cluster scheduling components occurs during training and the job container exits abnormally, the system unconditionally reschedules the job.

   Table 2 Rescheduling policy description

   Rescheduling Policy	Description	Supported Fault Types
   Direct rescheduling	The system isolates the faulty node or chip, and then directly reschedules the job.	Known node faults or chip faults at the rescheduling processing level.
   Unconditional retry	The system reschedules a job configured with unconditional retry within the specified number of times. After a successful rescheduling, the number of retry times decreases by 1. When the number of retry times reaches 0, rescheduling cannot be triggered again. To use the unconditional retry function, configure the fault-retry-times parameter in the YAML. For detailed parameter descriptions, see YAML Configuration Description.	Faults that cause jobs to exit abnormally and pod status to become Failed, which are caused by parameter plane network faults or training software faults.

Job-Level Rescheduling

If this mode is enabled, all Pods are stopped each time a fault occurs. After the faulty Pods are re-created and rescheduled, a training job is restarted. This mode is used by default.

For key configuration procedures of Job-level rescheduling, see Configuring Job-Level Rescheduling.

Constraints

• This feature is only supported in version 6.0.RC2 and later.
• In large-scale K8s cluster scenarios, ConfigMap mapping latency is uncontrollable. It is recommended to use shared storage for RankTable.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support Job-level rescheduling

Rescheduling Principles

If a software or hardware fault occurs during training, the training status becomes abnormal. Job-level rescheduling first destroys all training containers, isolates the faulty device, and then restarts and schedules training containers. Once restarted, the training process resumes from the beginning, similar to an initial training launch.

Figure 1 Principles

The following describes each step in the figure above.

1. After a fault is detected, first delete all Pods and containers of the current job.
2. Isolate the device where the fault is located to prevent it from being used again.
3. Recreate and reschedule the training Pods and containers.
4. After the container starts, restart the training process to resume training.

Pod-Level Rescheduling

If this mode is enabled, only the faulty Pods are stopped each time a fault occurs. After the faulty pods are re-created and rescheduled, a training job is restarted. If the fault cannot be rectified, Job-level rescheduling is triggered. Compared with Job-level rescheduling, Pod-level rescheduling reduces the time for resource scheduling and Pod creation.

For key configuration procedures of Pod-level rescheduling, see Configuring Pod-Level Rescheduling.

Constraints

• When Pod-level rescheduling is used for a training job in a large cluster, you are advised to set the open files parameter (maximum number of files that can be opened) to a large value. If the value is too small, pod rescheduling may be abnormal. For example, run the ulimit -n 100000 command to set open files to 100000.
• When a fault occurs on the Pod with hccl/rankIndex = 0 under annotation of a training job is faulty, pod-level rescheduling and process-level rescheduling are not triggered. Instead, Job-level rescheduling is triggered.
• Do not use ConfigMap to mount the RankTable file, as this may cause job rescheduling to fail.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that supports Pod-level rescheduling

Rescheduling Principles

If a software or hardware fault occurs during training, the training status becomes abnormal. Pod-level rescheduling destroys the faulty pods and training containers in the job, instructs the management processes in other training containers to destroy all training processes, isolates the faulty device, and reschedules and restarts training containers. Once restarted, management processes in all containers are notified to restart training processes to resume training.

1. After a fault is detected, only the faulty Pods and containers in the current job are deleted, and all training processes are destroyed.
2. Isolate the device where the fault occurs to prevent it from being used again.
3. Recreate and reschedule the training Pods and containers.
4. After the containers are started, restart the training processes to resume training.

Process-Level Rescheduling

This mode stops only the processes of the faulty node each time a fault occurs and determines whether to exit the faulty node based on the configured policy.

• recover policy: Migrate the containers on the faulty node to a healthy node.
• recover-in-place policy: For nodes where the following two types of faults occur, only the faulty processes are restarted, and the containers on the faulty node are not migrated. If faults occur on multiple nodes simultaneously, only the nodes with the following two types of faults will have their faulty processes restarted without container migration, while nodes with other fault types will have their containers migrated. If the fault types occurring on multiple nodes only include service process abnormal faults, containers on all faulty nodes will be migrated.
  • Service process abnormal fault.
  • Chip faults at the RestartRequest and RestartBusiness levels.

If recovery is not possible, it falls back to the job-level or pod-level rescheduling mode. Compared to Pod‑level rescheduling, this feature reschedules only the faulty process, significantly reducing the waiting time caused by asynchronous processes. It also leverages a new HCCL connection establishment scheme to greatly reduce connection setup time. Furthermore, it uses the high‑speed parameter‑plane P2P network between NPUs to transfer checkpoint information, avoiding the overhead associated with checkpoint saving and loading.

For the key configuration procedure of process-level rescheduling, see Configuring Process-Level Rescheduling.

Constraints

• For the PyTorch training framework, it must be used with a compatible MindSpeed version. For version compatibility, see MindSpeed-LLM.
• For the MindSpore training framework, it must be used with a compatible MindFormers version. For version compatibility, see MindSpore MindFormers.
• When a Pod with the hccl/rankIndex field set to 0 in the training job's annotation encounters a fault and the container needs to be migrated, Pod-level rescheduling and process-level rescheduling are not triggered; instead, Job-level rescheduling is triggered directly.
• Cannot be enabled simultaneously with graceful fault tolerance. If both are enabled, resumable training will recover training through Job-level rescheduling.
• In MindSpore scenarios, to ensure the normal use of this function, install MindSpore and MindIO in the same path.
• In the MindSpore scenario, due to framework mechanism limitations, process-level rescheduling carries a very low risk of failure.
• Do not mount RankTable files using ConfigMap, as this may cause job rescheduling to fail.
• PyTorch only supports single-operator mode and models based on the Megatron framework.
• Only training Ascend Jobs are supported.
• Only single-container migration is supported; affinity-based migration is not supported.
• Multi-modal models are not supported.
• The watchdog function is not supported.
• Process-level rescheduling triggered during checkpoint saving is not supported.
• For the Atlas A3 training series products, faults such as NPU removal or OS disconnection may cause process-level rescheduling to fail.
• When a fault occurs during the HCCL link setup phase, process-level rescheduling will fail. If, in addition to the HCCL link setup during training initialization, there are other HCCL link Setup phases during training, you can refer to the Configuring Proactive HCCL Link Setup section to establish links in advance to prevent faults from occurring during the HCCL link setup phase.
• It is not supported in IPv6 scenarios.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support process-level rescheduling

Rescheduling Principles

If a software or hardware fault occurs during training, the training status will become abnormal. Process-level rescheduling first destroys the faulty training process or container based on the configured policy, notifies the training processes in other training containers to pause the current training job, then isolates the faulty device, and reschedules and starts the training container again. After the faulty training container is restarted, it notifies the training processes in all containers to re-establish the collective communication link. After the link is established, the checkpoint is sent to the newly launched training process via the parameter plane to restore parameters. After restoration, all processes re-execute the current step to resume training.

Figure 1 Process-level rescheduling principles

The steps in the figure are described as follows:

1. After a hardware fault occurs on a device, the detection component of MindCluster on the server reports the fault information to ClusterD. Software faults are perceived by MindIO Controller in the container and reported to ClusterD.
2. ClusterD exits the faulty training process from the container on the faulty server and reschedules it to a standby server.
3. ClusterD notifies MindIO Controller on the master node to perform fault tolerance. The fault tolerance process includes notifying to stop training, notifying global faults, and notifying recovery policies.
4. MindIO Controller notifies MindIO Processor in each training process, and MindIO Processor calls PTA to forcibly stop the training process. MindIO Processor cleans up resources on normal nodes, destroys the communication domain, and waits for new processes to join after cleanup.
5. After the management process on the standby server starts the training process, a new MindIO Processor is created. MindIO Controller notifies MindIO Processor in each training process to resume training.
6. Each process establishes links through collective communication.
7. The NPU on the normal server transfers the checkpoint to the standby server through the parameter plane, and training continues after parameter state recovery is completed.

Feature Adaptation Points

In process-level rescheduling, the cluster brain decides the recovery policy based on global fault information and delivers the policy to MindIO. The scheduler needs to support scheduling of the faulty Pod rather than rescheduling the entire job, and support sequential fallback of recovery policies. In the training container, the framework first initializes the MindIO service. After the service is started, the optimizer reports the corresponding status to MindIO during updates. Subsequently, DP replica groups and optimizer replicas are created to ensure redundant backup of model parameters. When an exception occurs, the fault mode is captured by the exception capture decorator. During recovery, operator resource cleanup is performed, and communication re-establishment is triggered after the node restarts. Process-level rescheduling recovery is completed through online repair of the parameter plane and state rollback.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions as listed in Table 2.

Table 2 Functions adapted for process-level rescheduling

Process-Level Online Recovery

Process-level online recovery (also referred to as step-level recomputation recovery) is used to rectify the following faults:

• Network faults: Currently, only the following two scenarios are supported.

  • If BGP switches its link upon an HCCS L1-L2 port or link fault and operator-level online recovery fails, step-level recomputation is triggered to quickly rectify the fault without exiting processes. If operator-level online recovery is disabled, step-level recomputation is performed on training processes to rectify the fault without process interruption.
  • If operator-level online recovery fails to be executed upon an RoCE upper-level port or link fault, the training process is retried at the step level to quickly rectify the fault without exiting processes.

• On-chip memory faults: If an uncorrectable error (such as error 0x80E01801) occurs on the on-chip memory, the faulty on-chip memory space is isolated, and step-level recomputation is performed on training processes to quickly rectify the fault without exiting processes.

If faults cannot be rectified in the preceding two scenarios, rescheduling mode is then triggered.

Compared to process‑level rescheduling, process‑level online recovery does not reschedule the faulty process, significantly reducing the waiting time caused by asynchronous processes. At the same time, checkpoint information is transmitted through the high-speed parameter plane network P2P between NPUs, avoiding the time consumption of checkpoint saving and loading.

This fault handling mode is disabled by default. To enable it, see (Optional) Configuring Components.

For key configuration procedures of process-level online recovery, see Configuring Process-Level Online Recovery.

Constraints

• For the PyTorch training framework, it must be used with the MindSpeed version. For version compatibility, see MindSpeed-LLM.
• For the MindSpore training framework, it must be used with the MindFormers version. For version compatibility, see MindSpore MindFormers.
• This feature depends on the PyTorch memory management mechanism and can only be used when PYTORCH_NO_NPU_MEMORY_CACHING is not configured.
• This feature may not take effect in certain on-chip memory fault scenarios, such as memory address faults used by HCCL collective communication, which still require recovery through process-level rescheduling or higher-level fault tolerance solutions.
• For scenarios where faults occur in global variables defined in models or training scripts such as MindSpeed-LLM and MindSpeed, see FAQ for detailed handling policies.
• Cannot be enabled simultaneously with graceful fault tolerance. If both are enabled, checkpoint resume training will recover training through job-level rescheduling.
• In MindSpore scenarios, to ensure proper functionality, install MindSpore and MindIO in the same path.
• In MindSpore scenarios, set export TASKD_PROCESS_ENABLE="on" before starting the TaskD Manager.
• Do not mount the RankTable file using ConfigMap, as this may cause task rescheduling to fail.
• Multimodal models are not supported.
• MC2 enabled scenarios are not supported.
• The watchdog function is not supported.
• If a fault occurs during the HCCL link establishment phase, process-level online recovery will fail. If there are HCCL link setup phases in other training stages besides the initial training one, refer to the Configuring Proactive HCCL Link Setup section to establish links in advance, preventing faults from occurring during the HCCL link setup phase.
• IPv6 scenarios are not supported yet.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support process-level online recovery for network faults

Table 2 Products and frameworks that support process-level online recovery for on-chip memory faults

Process-level Online Recovery Principles

If an on-chip memory or network fault occurs during training, the training status will become abnormal. Process-level online recovery notifies all training processes to stop, retains the current training information, and rectifies the fault. Once recovery is complete, all training processes revert to the status at the end of the previous step. The healthy server transfers the checkpoint data to the affected server via the parameter plane to restore parameters. Training then resumes by re-executing the current step.

Figure 1 Process-level online recovery principles

The steps in the figure are described as follows:

1. After an on-chip memory fault or network fault occurs on a device, the detection component of MindCluster on the server reports the fault information to the cluster brain, ClusterD.
2. The on-chip memory fault or network fault is detected by CANN and reported to MindIO Processor and MindIO Controller through the training framework.
3. MindIO Controller requests a decision from the cluster brain on whether to perform step-level recomputation recovery. The cluster brain makes a decision based on the health status of other nodes in the cluster.
4. MindIO Controller notifies the MindIO Processor in each training process, invokes the training framework to stop the job, repair the fault, and retain the communication domain information.
5. The NPU on the normal server transfers the checkpoint to the faulty (repaired) server through the parameter plane, resumes training after restoring the parameter state, and restarts the current step computation.

Function Adaptation Points

In process-level online recovery, the cluster brain identifies network faults and on-chip memory faults based on fault information, issues corresponding recovery policies, and supports recovery policy rollback. In the training container, the framework first initializes the MindIO service. After the service is started, the optimizer reports the corresponding status to MindIO during updates. Subsequently, a DP replica group and optimizer replicas are created to ensure redundant backup of model parameters. When an exception occurs, the fault mode is captured through an exception capture decorator. During recovery, operator resource cleanup, UCE model optimizer reconstruction, parameter plane online repair, and state rollback are performed for different faults to complete process-level online recovery.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions on the framework.

Table 3 Functions adapted for process-level online recovery for network faults

Table 4 Functions adapted for process-level online recovery for on-chip memory faults

Operator-Level Online Recovery

Atlas A3 training series products support HCCL performing communication operator retransmission when a parameter plane network fault occurs. With the faulty process not exiting, operator-level online recovery can tolerate longer network anomalies without interrupting the training job.

If the operator-level online recovery (HCCL communication operator re-execution) for the network fault fails, it falls back to process-level online recovery.

For key configuration procedures of operator-level online recovery, see Configuring Operator-Level Online Recovery.

Usage Scenario

Currently, the operator-level online recovery function is supported in the following two fault scenarios.

• For chip network-related faults, if operator retransmission is successful, Volcano treats the current job as an unhealthy job. If operator retransmission fails, Volcano triggers rescheduling.
• For UnifiedBus device-related faults, after HCCL performs operator-level online recovery, Volcano treats the job as a sub-healthy job.

Constraints

• This feature does not support the scenario where MC2 is enabled.
• The watchdog function is not supported.

Supported Products and Frameworks

Table 1 Supported products and frameworks

Operator-Level Online Recovery Principles

Figure 1 Operator-level online recovery principles

The details of each step are as follows:

1. During training, a linkdown fault occurs on the HCCS or RoCE network plane.
2. CANN detects the network fault. Once the current operator is terminated, the system attempts to recover the network link by switching BGP links on the HCCS plane or by enabling link failover communication on the RoCE network plane. After recovery, the network operator is re-executed.
3. After the operator is re-executed successfully, the training iteration resumes.

Suspension and Switchback of Link Failover Communication

For Atlas A3 training series products, MindCluster cluster scheduling components provide suspension and switchback functions for link failover communication of training jobs, allowing you to freely switch RoCE network ports used by NPUs during training via active link failover and switchback interfaces.

To learn about networking relationships of NPUs when this feature is used, see "Network Plane Introduction > Parameter Plane Network > Port Interconnection Policy" section of the Ascend Training Solution Networking Guide (Atlas A3 Training Product).

For details about how to configure suspension and switchback of link failover communication, see Configuring Suspension and Switchback of Link Failover Communication.

• Before calling the link failover and switchback APIs to perform link failover and switchback, understand NPU networking first and ensure that the network link of the target NPU is normal. If the target NPU is in the linkdown state, the operation fails.

• The following uses the interface interconnection in the networking guide as an example to describe the dev-op mapping when the SwitchNicTrack API is called.

  1. If device 0 and device 8 are switched from QDD8 to QDD7, dev should be [device0, device8] and op should be [true, true].
  2. If device 0 and device 8 are switched back from QDD7 to QDD8, dev should be [device0, device8] and op should be [false, false].
  3. If device 0 is switched from PortA of QDD8 to PortA of QDD7, dev should be [device0] and op should be [true].
  4. If device 0 is switched back from PortA of QDD7 to PortA of QDD8, dev should be [device0] and op should be [false].
  5. If devices of leaf 1 are switched to leaf 2, dev should be [device0, device8, device2, device10, device4, device12, device6, device14] and op should be [true, true, true, true, true, true, true, true].
  6. If all devices of leaf 2 are switched back to leaf 1, dev should be [device0, device8, device2, device10, device4, device12, device6, device14] and op should be [false, false, false, false, false, false, false, false].

  Figure 1 Port interconnection relationship

Usage Scenario

Currently, this feature can be used in the following two scenarios:

• Switch upgrade: Link failover is manually triggered to upgrade switches. After that, links are switched back.
• Troubleshooting: After the faulty port where link failover occurs is recovered, manually switch back links.

Constraints

• Issue link failover or switchback command after the training iteration is normal.
• Ensure that process-level recovery is enabled.
• Currently not supported in IPv6 scenarios.
• Only supports RoCE communication between Pods.

Supported Products and Frameworks

Table 1 Supported products and frameworks

Principles of Suspension and Switchback of Link Failover Communication

Figure 2 Principles

The details of each step are as follows:

1. The AI platform integrates ClusterD and calls the gRPC interface of ClusterD to issue a failover operation, specifying the NPUs to be switched.
2. ClusterD notifies MindIO to pause training.
3. TaskD Manager notifies all TaskD Workers to call the training framework interface to perform the failover operation.
4. The training framework calls the CANN interfaces one by one according to the communication domain to perform the failover operation.
5. After ClusterD determines that the failover operation for all NPUs is complete, TaskD notifies MindIO to continue executing the next training step after the switchover.

Function Adaptation Points

During suspension and switchback of link failover communication, the framework initializes the MindIO service. After the service is started, the optimizer updates the corresponding status to MindIO. The graceful suspension mechanism is called for job suspension and failover. The cluster brain needs to provide an external interface to receive failover instructions and manage the link failover communication process.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions as listed in Table 2.

Table 2 Functions adapted for suspension and switchback of link failover communication

(Optional) Graceful Fault Tolerance

You can enable graceful fault tolerance if no backup resources are available for training jobs or if you expect a device to automatically recover. That is, if a processor is faulty during training, the system attempts to automatically recover the faulty processor. If it can be recovered, the system starts the job to continue the training while the pod is still running. If the fault persists, the system rolls back to the rescheduling mode.

Graceful fault tolerance can automatically recover the faulty device without resource scheduling. However, it cannot reduce the recovery time during training initialization. Generally, the recovery time required by graceful fault tolerance is longer than that required by process-level rescheduling and process-level online recovery.

To understand the key configuration procedure for graceful fault tolerance, see Configuring Graceful Fault Tolerance.

Constraints

• Currently, graceful fault tolerance is only supported for chip faults.
• Graceful fault tolerance cannot be enabled simultaneously with process-level rescheduling or process-level online recovery. If both are enabled, resumable training will recover the training through Job-level rescheduling.
• IPv6 scenarios are not supported yet.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support graceful fault tolerance

Graceful Fault Tolerance Principles

If rescheduling is triggered during node or processor fault handling, O&M personnel need to manually restore the faulty device. If it is not restored in a timely manner, a large number of scattered faults may occur in a training cluster, reducing the cluster computing power utilization. Therefore, graceful fault tolerance is added for resumable training to optimize the fault tolerance capability of NPUs for some faults.

These NPU faults can be rectified by exiting the training processes and performing hot resets on the NPUs. The graceful fault tolerance mode is designed to handle such faults and does not require job rescheduling.

Ascend Device Plugin reports faults and recovers devices. The management process (Elastic Agent for PyTorch and TaskD for MindSpore) stops and restarts training processes based on the information reported by Ascend Device Plugin to complete fault recovery. If faults cannot be recovered, the rescheduling mode is used again. To integrate the graceful fault tolerance mode, add a management process to the service container. The management process must have the capabilities of detecting faults, stopping training jobs, and restarting training jobs.

In graceful fault tolerance mode, a fault is directly reported to the management process in the service container (usually by mounting a file). The management process in the container then reads the fault file to obtain specific fault information. The process of obtaining fault information is shown in Figure 1.

Figure 1 Obtaining fault information

Faults are classified into four types in graceful fault tolerance mode: no handling required, service re-execution required, processor reset required, and rescheduling required. The handling for each fault type is shown in Figure 2.

Figure 2 Graceful fault tolerance fault handling process

Online Stress Testing

MindCluster supports online stress tests during training. That is, you can call the online stress testing interface to suspend a specified training job and perform hardware P2P or AIC stress tests on the nodes running the job. If no fault exists, training resumes. If a fault exists, the faulty node is isolated and resumable training is triggered.

Constraints

• For the PyTorch training framework, MindSpeed-LLM 2.3.0 is required. For version compatibility, see MindSpeed-LLM.
• For the MindSpore training framework, MindFormers master version is required. For version compatibility, see MindSpore MindFormers.
• Issue online stress testing commands only after training has entered normal iteration.
• Ensure that process-level recovery features are enabled.
• Restarting ClusterD is not supported during stress testing. If ClusterD restarts abnormally, you need to restart the training and re-issue the stress testing task.
• The hot reset function must be disabled during stress testing.
• For P2P stress testing, ensure that the device side has more than 10 GB of free memory.
• You need to add the nodeDEnable=on label to the node to ensure that the node undergoing stress testing can be isolated.
• For the MindSpore training framework, you need to set export TASKD_PROCESS_ENABLE="on" before starting the TaskD Manager.
• Usage in IPv6 scenarios is not supported currently.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support online stress testing

Online Stress Testing Principles

Figure 1 Schematic diagram

The details of each step are as follows:

1. The AI platform integrates ClusterD, calls the gRPC interface of ClusterD to deliver stress testing operations, and specifies the nodes to be tested.
2. ClusterD notifies MindIO to suspend training.
3. TaskD Manager notifies the specified TaskD Worker to call the training framework interface to perform stress testing.
4. The training framework calls the CANN interface on the specified NPU to perform stress testing.
5. After ClusterD determines that the stress testing on the specified NPU is complete, TaskD notifies MindIO to continue executing the next training step after the stress testing is finished.

Function Adaptation Points

During online stress testing, the framework initializes the MindIO service. After the service is started, the optimizer updates the corresponding status to MindIO. The graceful suspension mechanism is called for job suspension. After the suspension, a hardware stress test is performed. After the test is complete, training continues. The cluster brain needs to provide an external interface to receive stress test instructions and manage the stress test process.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions as listed in Table 2.

Table 2 Functions adapted for online stress testing

Hot Switching

After the hotSwitch policy is configured for a training job, if a subhealth fault occurs, the training process is paused after the backup node is started, and then the training job is restarted using the backup node.

Constraints

• For the PyTorch training framework, it must be used with MindSpeed-LLM version 2.3.0. For version compatibility, see MindSpeed-LLM.

• For the MindSpore training framework, it must be used with the MindFormers master version. For version compatibility, see MindSpore MindFormers.

• Only supports PyTorch single-operator mode, models based on the Megatron framework, and training Ascend Jobs.

• In MindSpore scenarios, to ensure the normal use of this feature, install MindSpore and MindIO in the same path.

• Multimodal models are not supported.

• The watchdog function is not supported.

• If a hot switchover is triggered before the training task produces an iteration, it may cause MindIO to block, ultimately triggering a job-level rescheduling.

• Hot switching is not supported if the pod annotated with hccl/rankIndex = 0 in a training job is subhealthy.

• If any of the following exceptions occurs, Job-level rescheduling is triggered, and the subhealth node handling policy is downgraded to ignore, meaning that subhealth faults are not handled.

  • After the backup Pod is started, training suspension fails.
  • After the backup Pod is started, MindCluster times out (15 minutes) waiting for the training suspension status to be reported.
  • The backup Pod fails to run.
  • After the original Pod is deleted, training recovery fails.
  • After the original Pod is deleted, MindCluster times out (15 minutes) waiting for the training recovery status to be reported.

• After the hotSwitch policy is configured, the process-level recovery option is automatically added. If a non-subhealth fault occurs, process-level recovery is triggered.

• In a scenario without standby nodes, the hot switching process cannot be completed. In this case, the subhealth fault handling policy is degraded to ignore, and subhealth faults are not handled.

• Currently not supported in IPv6 scenarios.

Supported Product Models and AI Frameworks

Table 1 Products and frameworks that support hot switching

Hot Switching Principles

Figure 1 Schematic diagram

The details of each step are as follows:

1. ClusterD detects a sub-health fault through Ascend Device Plugin.
2. ClusterD decides whether to perform hot switching based on the configured policy.
3. ClusterD notifies Ascend Operator to start the backup Pod.
4. Volcano schedules the backup Pod.
5. A new MindIO Processor is created in the backup Pod, and MindIO Processor initiates registration with MindIO Controller.
6. MindIO Controller sends a training suspension notification.
7. MindIO Controller notifies ClusterD that training is paused.
8. ClusterD notifies Volcano to delete the faulty Pod.
9. ClusterD notifies MindIO to resume training.

Function Adaptation Points

During hot switching, the cluster brain sets annotations for the faulty pod based on the subhealth fault information, starts and schedules the backup pod, and notifies MindIO of the hotSwitch policy. Training resumes after it is switched to the backup pod. In the training container, the framework initializes the MindIO service. After the service is started, the optimizer updates the corresponding status to MindIO. When an exception occurs, the decorator is used to capture fault modes. After a new node is started, training on the normal node is paused. Then, the communicator is rebuilt, the parameter plane of the new node is restored, and the node hot switching is complete after training is complete.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions as listed in Table 2.

Table 2 Functions adapted for hot switching

Elastic Training

When a hardware fault occurs and there are no available backup resources in the K8s cluster, MindCluster first scales down some nodes according to the data parallel domain to continue training. When idle resources become available in the cluster, it triggers scale-up to restore the original training scale. Compared with process-level rescheduling, this solves the problem of no available backup resources in the cluster for rescheduling.

Constraints

• Only supports PyTorch with MindSpeed-LLM 2.3.0. For version compatibility, see MindSpeed-LLM.

• Only supports acjob type training tasks.

• Depends on the optimizer replica of MindIO. A full optimizer replica must exist, so MindIO and TaskD need to be installed and used together.

• Cannot be enabled simultaneously with the graceful fault tolerance feature.

• Elastic training cannot be triggered when a fault occurs on the Pod whose hccl/rankIndex field is 0.

• Multimodal models are not supported.

• The watchdog feature cannot be enabled.

• Since elastic training creates additional communication groups, it may increase on-chip memory usage.

• Currently not supported in IPv6 scenarios.

  Memory size calculation formula: Maximum increased memory (MB) = HCCL_BUFFSIZE * 2 * 9, where HCCL_BUFFSIZE defaults to 200 MB. For details about HCCL_BUFFSIZE, see the "HCCL_BUFFSIZE" section in the CANN Environment Variable Reference.

For more constraints, see MindSpeed-LLM Elastic Training Constraints.

Supported Product Types and AI Frameworks

Table 1 Products and frameworks that support elastic training

Elastic Training Principles

Figure 1 Schematic diagram

In the figure, only one DP domain is scaled in. In actual elastic training, multiple DP domains may be scaled in at a time. Each square in the figure represents a rank.

1. Distributed training is performed normally according to TP (Tensor Parallelism), PP (Pipeline Parallelism), and DP (Data Parallelism).
2. At a certain point during training, if a rank fault occurs and no more idle resources are available in the cluster for resumable training, the DP domain is scaled in, meaning the Pod corresponding to one DP domain (which may include multiple Pods) is scaled in, and training continues.
3. At a certain moment during scale-in training, if idle resources are available in the cluster, the removed pods are rescheduled, and the cluster is scaled out to the original scale for further training.

Figure 2 Elastic training flowchart

The details of each step are as follows:

1. After a hardware fault occurs on a device, the detection component of MindCluster on the server reports the fault information to ClusterD. Software faults are detected by MindIO Controller inside the container and reported to ClusterD.
2. ClusterD destroys the container on the faulty server.
3. If no backup node is available to schedule a new container, ClusterD notifies MindIO Controller on the master node to perform scale-in training.
4. MindIO Controller notifies MindIO Processor in each training process, and MindIO Processor calls PTA to stop the training process and clean up resources on normal nodes.
5. MindIO Controller notifies MindIO Processor in normal training processes to execute scale-in procedures such as communication group reconstruction and perform scale-in training.
6. Detect that the Pod deleted during scale-in has been rescheduled successfully.
7. ClusterD notifies MindIO Controller through TaskD Manager to perform scale-out.
8. MindIO Controller notifies MindIO Processor in each training process, and MindIO Processor calls PTA to stop the training process and clean up resources on normal nodes.
9. Each process establishes links through collective communication.
10. NPUs of the normal server transfer the checkpoint data to the standby server through the parameter plane. After the parameter status is restored, the training continues.

Adaptation Feature Points

In elastic training, the cluster brain decides on a recovery policy based on global fault information and delivers the policy to MindIO. The scheduler must support scheduling of faulty Pods rather than rescheduling the entire job, and support sequential fallback of recovery policies. In the training container, the framework first initializes the MindIO service. After the service is started, the optimizer reports the corresponding status to MindIO during updates. Subsequently, DP replica groups and optimizer replicas are created to ensure redundant backup of model parameters. When an exception occurs, the fault mode is captured by the exception capture decorator and reported by MindIO to the cluster brain for decision-making.

• When the cluster brain detects a fault and no redundant backup resources are available, it delivers a scale-in policy to MindIO, which performs operator resource cleanup, scale-in reconstruction, and continues training in the scaled-in state.
• When the cluster brain detects available resources and a new node is successfully brought up, it delivers a scale-out policy to MindIO, which performs operator resource cleanup, scale-out communication reconstruction, scale-out parameter plane recovery, and scale-out state rollback, completing elastic scale-out to restore the original scale and continue training.

For non-MindSpeed-LLM/MindCluster users, adapt the following functions listed in Table 2..

Table 2 Functions adapted for elastic training

The adaptation items numbered 1 to 6 in Table 2 are common logic for MindIO TFT (MindCluster MindIO Training Fault Tolerance), and the adaptation items numbered 17 to 18 are common logic for resumable training. They are not described in detail in this chapter. The following briefly introduces the unique functional points of elastic training based on Megatron 0.12.1.

• Elastic training callback registration

  Enabled during training startup initialization, it registers the callback functions that need to be executed during elastic training scale-in and scale-out recovery into MindIO, so that they can be invoked during the recovery process.

• Scale-in rebuilding

  1. Create a new global communication group based on the members after scale-in and record it, which will subsequently replace the original global communication group for communication.
  2. Record the original framework parameters such as DP size and num_microbatches for subsequent scale-out recovery, and update them with the data after scale-in.
  3. Rebuild other local communication groups after scale-in based on the faulty rank information, and update the communication groups in instance objects such as models and optimizers.
  4. Rebuild the dataset, and reinitialize some framework instances and parameters.

• Scale-out communication rebuilding

  1. Rebuild the global and local communication groups after scale-up, and update the communication groups in instance objects such as models and optimizers.
  2. Restore framework parameters such as DP size, and reinitialize some framework instances.

• Scale-out parameter plane recovery

  1. Create communication groups for the newly launched rank training processes and the backup rank training processes, used for sending and receiving optimizer parameters and the like.
  2. The backup rank training processes send the optimizer parameters required for recovery to the newly launched rank training processes.
  3. After receiving the optimizer parameters, the newly launched rank training processes update parameters such as optimizer, opt_param_scheduler, and global args as needed.

• Scale-out state rollback

  1. Restore framework parameters such as num_microbatches.
  2. Before resuming training, copy the optimizer parameters to the model parameters, and perform an all_gather communication operation within the corresponding DP domain to ensure the model parameters are in the latest state.
  3. Fix the printing of training iteration logs.
  4. Rebuild the dataset and reinitialize some framework instances, parameters, etc.
  5. Destroy the communication groups used for sending and receiving parameters during the recovery process.

• Torch communication adaptation for newly launched nodes

  1. For the restarted node, communication operators are issued from the pretrain startup process to the entry into training. However, the normal training rank does not rebuild the communication domain in coordination with the restarted node at this stage, so collective communication cannot succeed. Therefore, it is skipped directly.
  2. For the restarted node, a parallel communication domain is created from the pretrain startup process to the entry into training. However, the normal training rank does not rebuild the communication domain in coordination with the restarted node at this stage, which causes errors for the gloo group. Therefore, the creation of a new gloo communication group is skipped directly.

• Scale-in training global group communication adaptation

During scale-in training, because the faulty node has been removed, communication using the original global communication group will fail. It must be replaced with the scaled-in global communication group.

• Scale-in training replica group communication adaptation

In the LLM repository reference link, start_param_sync_wrapper, get_grad_norm_fp32_wrapper, get_parameter_state_dp_zero_wrapper, etc., are patched to adapt replica group communication during scale-in training. The following uses get_parameter_state_dp_zero_wrapper as an example to introduce the replica group adaptation principles:

Assume tp=8, pp=1, and dp=4. The DP groups are ranks [0,8,16,24], [1,9,17,25], [2,10,18,26], …, [7,15,23,31] respectively. According to the replica optimizer principle, the replica groups are ranks [0,8], [16,24], [1,9], [17,25], [2,10], [18,26], …, [7,15], [23,31] respectively, with rank 0-15 and rank 16-31 being replicas of each other. After rank 31 fails, the DP domain corresponding to rank 24-31 is removed to continue scale-in training.

Native Megatron uses the group corresponding to the data_parallel_group_gloo member variable of the optimizer instance (i.e., the DP group, which is the replica group when using MindIO's optimizer replica) for communication. After scale-in, the replica groups that do not include the removed rank 24-31 continue to communicate using the original communication groups. The replica groups that include the scaled-in ranks use a scaled-in group composed of the normal rank within the group and the replica rank corresponding to the scaled-in rank for communication. For example, after the replica group rank [23,31] is scaled in, the communication group used for communication is rank [23,15].

• Scale-in training parameter adaptation

  In LLM Repository Reference Link, functions such as patch_world_size_func_wrapper, log_wrapper, is_last_rank_wrapper,optimizer_param_scheduler_step_wrapper,track_app_tag_wrapper,print_rank_last_wrapper, and num_floating_point_operations_wrapper are patched to adapt parameters used during training, such as global_batch_size and world_size. For example: the native implementation uses dp_size*micro_batch_size*num_microbatches, but after scaling in, num_microbatches may differ across DPs, so args.globatch_size is used directly. After scaling in, the global group after scaling in is used to determine whether it is the last rank; the global group size is modified to the size after scaling in, etc.

• Gradient precision calculation

  In LLM Repository Reference Link 1, start_grad_sync_wrapper, forward_step_wrapper, and elastic_training_get_forward_backward_func_wrapper, as well as the loss\_func code pointed to by LLM Repository Reference Link 2, are patched or modified to adapt to precision gradient changes caused by scaling in.

  • loss_func is modified from performing all_reduce communication within the DP group for each micro_batch to not performing communication during scaling-in training. The reason is that after scaling in, the number of num_micro_batches within each DP domain may differ, causing the first few DPs to execute one extra all_reduce and become stuck.
  • In start_grad_sync_wrapper, gradient_scaling_factor is modified to 1.0 / (arguments.global_batch_size / arguments.micro_batch_size), i.e., dividing by num_micro_batches on top of the original 1/dp_size.
  • forward_step_wrapper changes num_microbatches to 1, so that the loss calculation no longer divides by num_microbatches, because it has already been divided by num_microbatches in start_grad_sync_wrapper.
  • In elastic_training_get_forward_backward_func_wrapper, because loss_func does not perform all_reduce within the DP group, after the native forward_backward_func completes, at the last PP stage, the sum of each key in losses_reduced (i.e., the sum of lm loss across all micro_batches) performs an all_reduce sum operation within the DP group.

Recovery Acceleration

Training Recovery Principles

After a fault is rectified, the training process is restarted. The started training process needs to save and load model weights to return to the training state when a job is interrupted. During normal training, checkpoint files of the training model weights are saved at regular intervals. The training process restarted after training termination can load the previously saved checkpoint file to restore the model weight state at a certain checkpoint, thus reducing training time. The methods of saving and loading checkpoints vary according to frameworks. The following lists some examples of saving and loading checkpoints for TensorFlow, PyTorch, and MindSpore. You can modify your training model script based on these examples.

PyTorch

1. Save the checkpoint.

   def save_checkpoint(state, is_best, args, filename='checkpoint.pth.tar'):
       filename2 = os.path.join(args.save_ckpt_path, filename)
       torch.save(state, filename2)
       if is_best:
           shutil.copyfile(filename2, os.path.join(args.save_ckpt_path, 'model_best.pth.tar'))

2. Load the checkpoint.

   checkpoint = torch.load(args.checkpoint_path, map_location=loc)
   args.start_epoch = checkpoint['epoch']
   best_acc1 = checkpoint['best_acc1']
   model.load_state_dict(checkpoint['state_dict'])
   optimizer.load_state_dict(checkpoint['optimizer'])

MindSpore

1. Save the checkpoint.

   ms.save_checkpoint(net, "./lenet.ckpt",
                      choice_func=lambda x: x.startswith("conv") and not x.startswith("conv1"))

2. Load the checkpoint.

   param_dict = ms.load_checkpoint("./lenet.ckpt")

Periodic Checkpoint Saving

Currently, training data (such as model parameters) is saved as checkpoints to implement large-scale cluster training. When a service platform detects a fault, it can terminate the current training job and reload the saved checkpoints to resume training from the time when checkpoints are saved, avoiding a complete restart.

Periodic checkpoint saving consists of two parts: asynchronous checkpoint saving and memory checkpoint loading.

• Asynchronous checkpoint saving

  MindIO ACP provides the capability of asynchronous saving checkpoints at a fixed interval. If MindIO ACP is not used, the parameters to be saved need to be copied from the device to the host and then flushed to storage, which takes several minutes. MindIO ACP enables asynchronous flushing, allowing parameters to be written to storage in the background after being copied to the host from the device, without blocking the ongoing training process. This allows training to proceed uninterrupted during the flushing phase.

• In-memory checkpoint loading

  MindIO ACP provides the capability of periodically loading checkpoints based on memory. During training recovery, periodic checkpoints that are saved previously need to be loaded from storage to restore the training status and resume training. However, checkpoint loading within a large model typically takes several minutes due to data volume and storage performance constraints. To accelerate this process, MindIO ACP introduces a periodic memory-based checkpoint loading mechanism. In the event of a fault, checkpoints are loaded directly from memory, significantly reducing recovery time.

Recommended Configuration

When using the checkpoint saving capability of the rescheduling upon faults feature, select a frequency for periodically saving checkpoints based on your actual requirements. Figure 1 illustrates recommended frequencies.

Figure 1 Recommended frequencies for periodic checkpoint saving

When periodic checkpoint recovery is enabled, any training progress between the last saved checkpoint and the fault event will be lost upon recovery. To minimize this loss, you can reduce the interval between checkpoint saving. However, each saving operation interrupts training while checkpoints are flushed from the device to storage, incurring training time wastes. As a result, shorter intervals lead to wasted training time and status loss. Therefore, assuming the time to save checkpoints remains constant, a trade-off must be made between minimizing training loss and avoiding loss caused by faults.

To solve this problem, the single saving time needs to be reduced. However, this duration is largely influenced by the volume of data being saved and the performance of the storage system, both of which are typically difficult to optimize. Therefore, MindIO ACP is introduced to solve the problem of high loss during periodic checkpoint recovery.

Dying Gasp Checkpoint Saving

While asynchronous checkpoint saving minimizes the checkpoint interval and fault-related loss, it still incurs overhead, making sub-second fault loss reduction challenging. MindCluster introduces dying gasp checkpoint saving, preserving the initial parameter state of the current step upon fault occurrence, effectively reducing status loss to less than one step.

MindCluster MindIO Try To Persist (MindIO TTP for short) provides the dying gasp checkpoint capability, enabling users to preserve dying gasp checkpoints when a fault occurs.

For details about how to save dying gasp checkpoints, see Fault Recovery Acceleration.

For details about how to configure dying gasp checkpoint saving, see Configuring Dying Gasp Saving.

Function Adaptation Points

In dying gasp checkpoints, the framework initializes the MindIO service. After the service is started, the optimizer updates the corresponding status to MindIO. Then, a DP replica group and optimizer replicas are created to ensure redundant backup of model parameters. When an exception occurs, the decorator is used to capture fault modes. Then, operator resources are cleared, and dying gasp checkpoints are saved based on replicas.

For non-MindSpeed-LLM users, the functional adaptations in Table 1 must be completed on the framework side.

Table 1 Functions adapted for dying gasp checkpoint saving

Restoring Checkpoint Transmission on the Parameter Plane

With the dying gasp checkpoint mechanism, training rollback loss is reduced to a single step. However, during a fault, checkpoints still need to be flushed to storage drives, and upon fault tolerance and training resumption, must be reloaded, which prolongs the total recovery time. To address this, MindCluster introduces checkpoint transmission and recovery over the parameter plane.

When a fault occurs, the parameter state is retained in the device. Once fault tolerance is complete, the parameter state from the healthy card is transmitted to the fault-tolerant card via the parameter plane network, enabling rapid parameter restoration on the fault-tolerant card. Currently, this capability must be used together with process-level rescheduling and process-level online recovery.

To learn about the checkpoint configuration procedure on the parameter plane, see Restoring Parameter Passing on the Parameter Plane.