Using via Command Line

(Optional) Configuring Components

If you have already configured resumable training-related functions when installing Ascend Device Plugin and NodeD, you can skip this chapter. If not, you need to configure Ascend Device Plugin and NodeD accordingly.

Configuring Ascend Device Plugin

Only image-based startup of Ascend Device Plugin is supported.

Modify the startup YAML of Ascend Device Plugin based on the fault handling mode used. The bold parts below indicate the modifications.

Rescheduling mode

In rescheduling mode, an exception in Ascend Device Plugin also triggers fault rescheduling.

 ...
       containers:
       - image: ascend-k8sdeviceplugin:v{version}
         name: device-plugin-01
         resources:
           requests:
             memory: 500Mi
             cpu: 500m
           limits:
             memory: 500Mi
             cpu: 500m
         command: [ "/bin/bash", "-c", "--"]
         args: [ "device-plugin
                  -useAscendDocker=true
                  -volcanoType=true                    # Volcano must be used in rescheduling scenarios
                  -autoStowing=true                    # Whether to enable the automatic management switch. The default value is true. Setting it to false disables automatic management. When the chip health status changes from unhealthy to healthy, it will not be automatically added to the scheduled resources pool. When automatic management is disabled, after the chip parameter plane network fault recovery, it will not be automatically added to the scheduled resources pool. This feature is only applicable to Atlas training series products.
                  -listWatchPeriod=5                   # Set the health status check period, in seconds. Range: [3,1800]
                  -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log
                  -logLevel=0" ]
         securityContext:
           privileged: true
           readOnlyRootFilesystem: true
 ...

(Optional) Graceful fault tolerance mode: Based on the rescheduling configuration, add the -hotReset field.

The graceful fault tolerance feature has been sunset. It will not be supported in PyTorch versions beyond 7.2.RC1 and MindSpore versions beyond 7.1.RC1.
The function corresponding to -hotReset = 1 has been sunset.

 ...
       containers:
       - image: ascend-k8sdeviceplugin:v{version}
         name: device-plugin-01
         resources:
           requests:
             memory: 500Mi
             cpu: 500m
           limits:
             memory: 500Mi
             cpu: 500m
         command: [ "/bin/bash", "-c", "--"]
         args: [ "device-plugin
                  -useAscendDocker=true
                  -volcanoType=true                    # Volcano must be used in rescheduling scenarios
                  -autoStowing=true                    # Whether to enable the automatic management switch. The default value is true. Setting it to false disables automatic management. When the chip health status changes from unhealthy to healthy, it will not be automatically added to the scheduled resource pool. When automatic management is disabled, the chip will not be automatically added to the scheduled resource pool after the parameter plane network fault recovery. This feature is only applicable to Atlas training series products.
                  -hotReset=1 # Enable graceful fault tolerance mode. The system will attempt to automatically reset faulty chips.
                  -listWatchPeriod=5                   # Health status check period, in seconds. Range: [3,1800]
                  -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log
                  -logLevel=0" ]
         securityContext:
           privileged: true
           readOnlyRootFilesystem: true
 ...

Run the following command on the K8s management node to start Ascend Device Plugin.
```
kubectl apply -f device-plugin-xxx-v{version}.yaml
```
The following is an example of starting this component in an Atlas training environment:
```
kubectl apply -f device-plugin-volcano-v{version}.yaml
```

Configuring NodeD

You can configure the interval for reporting node status by manually modifying the NodeD startup YAML.

Go to the component decompression directory and run the following command to open the startup YAML file of NodeD.
```
vi noded-v{version}.yaml
```

Modify the -reportInterval parameter in the args line of the YAML file, as shown below:

 ...
           env:
             - name: NODE_NAME
               valueFrom:
                 fieldRef:
                   fieldPath: spec.nodeName
           imagePullPolicy: Never
           command: [ "/bin/bash", "-c", "--"]
           args: [ "/usr/local/bin/noded -logFile=/var/log/mindx-dl/noded/noded.log -logLevel=0 -reportInterval=5" ]
           securityContext:
             readOnlyRootFilesystem: true
             allowPrivilegeEscalation: true
           volumeMounts:
             - name: log-noded
 ...

Building an Image

Building a MindSpeed-LLM Training Image (PyTorch)

MindSpeed-LLM, as the Ascend large model training framework, aims to provide an end-to-end large language model training solution for Ascend chips, including distributed pre-training, distributed instruction fine-tuning, distributed preference alignment, and the corresponding development toolchain. The MindSpeed-LLM User Guide includes chapters on repository cloning, environment setup, and large model training. To build a MindSpeed-LLM training framework image, you can refer to this section and the MindSpeed-LLM User Guide.

Resumable training can be built based on the base training image. For details on building the base training image, see the Building a Container Image Using Dockerfile (PyTorch) section.

This section demonstrates how to build a training image based on Ubuntu 20.04, incorporating the steps for building the base training image.

The following example uses MindSpeed-LLM version 2.3.0.

Preparing Software Packages

As shown in Table 1,obtain the software packages of the corresponding OS and prepare the Dockerfile and script file required by the image. In the package names, {version} indicates the version number, {arch} indicates the architecture, and {chip_type} indicates the chip type.

Table 1 Preparing software packages

Software Package	Required	Description	Obtaining Method
taskd-{version}-py3-none-linux_{arch}.whl	Yes	Resumable training whl package Note: Before installing TaskD, ensure that the PyTorch framework is correctly installed. The currently supported PyTorch versions are 2.1.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.1. TaskD depends on the PyTorch framework. Select a PyTorch version without known security vulnerabilities or obtain the corresponding version with security issues fixed from the official community.	Download Link Note: The link points to the download page of the TaskD's compressed package Ascend-mindxdl-taskd_{version}_linux-{arch}.zip. You need to decompress it to obtain the corresponding whl software package.
mindio_ttp-{version}-py3-none-linux_{arch}.whl	Yes	MindIO TFT installation package.	Download Link
apex-0.1+ascend-cp3x-cp3x-linux_{arch}.whl	Yes MindSpeed-LLM dependency	Mixed precision training uses a mix of single-precision (float32) and half-precision (float16) data types during training, combining both and using the same hyperparameters to achieve almost the same precision as float32. The cp3x in the software package indicates the Python version number. For example, x being 10 indicates Python 3.10. The specific Python version is subject to the MindSpeed-LLM version.	For details, see the "Installing the APEX Module" section in Ascend Extension for PyTorch Installation Guide, and compile the APEX package based on the actual situation.
torch_npu-2.7.1.{version}-cp3x-cp3x-manylinux_2_28_{arch}.whl	Yes MindSpeed-LLM dependency	Ascend Extension for PyTorch is a deep learning adaptation framework based on Ascend, enabling Ascend NPUs to support the PyTorch framework, providing PyTorch users with the superior computing power of Ascend AI processors. The cp3x in the software package indicates the Python version number. For example, x being 10 indicates Python 3.10. The specific Python version is subject to the MindSpeed-LLM version.	Download Link Note: To use the PyTorch model from the MindSpeed-LLM code repository, Ascend Extension for PyTorch 2.6.0 or later is required.
x86_64 architecture: torch-2.7.1+cpu.cxx11.abi-cp3x-cp3x-linux_x86_64.whl ARM architecture: torch-2.7.1+cpu-cp3x-cp3x-manylinux_2_28_aarch64.whl	Yes MindSpeed-LLM dependency	Official PyTorch package. The cp3x in the software package indicates the Python version number. For example, x being 10 indicates Python 3.10. The specific Python version is subject to the MindSpeed-LLM version.	Download Link
Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run	Yes For versions before CANN 8.5.0, this package is named Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run	CANN operator package.	Download Link Note: Obtain the software package that matches the server.
Ascend-cann-toolkit_{version}_linux-{arch}.run	Yes	CANN Toolkit package.	Download Link Note: Obtain the software package that matches the server.
MindSpeed	Yes	MindSpeed is a large model acceleration library for Ascend devices.	git clone https://gitcode.com/Ascend/MindSpeed.git cd MindSpeed git checkout 2.3.0_core_r0.12.1
version.info	Yes Dependency file for installing CANN.	Driver version information file.	Copy /usr/local/Ascend/driver/version.info from the host.
ascend_install.info	Yes Dependency file for installing CANN.	Driver installation information file.	Copy /etc/ascend_install.info from the host.
Dllogger code repository	Yes	PyTorch log tool.	git clone https://github.com/NVIDIA/dllogger.git
get-pip.py	Yes	Used to install the pip module.	curl -k https://bootstrap.pypa.io/get-pip.py -o get-pip.py
Dockerfile	Yes	Used to image creation.	-

To prevent software packages from being maliciously tampered with during transmission or storage, you need to download the corresponding digital signature file for integrity verification when downloading software packages.

After downloading the software package, see the OpenPGP Signature Verification Guide to perform PGP digital signature verification on the software package downloaded from the Support website. If the verification fails, do not use the software package and contact Huawei technical support engineers for resolution.

Before installing or upgrading using a software package, you must also verify the digital signature of the software package following the above process to ensure that the software package has not been tampered with.

For carrier customers, please visit: https://support.huawei.com/carrier/digitalSignatureAction

For enterprise customers, please visit: https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054

This section uses a single Atlas 800T A2 training server, Ubuntu 20.04 Arm, and Python 3.10 as an example to describe how to build a training image. Modify the relevant steps based on the actual situation during use.

Procedure

Prepare the software packages on the host by referring to Table 1.

Write the following Dockerfile.

FROM ubuntu:20.04
WORKDIR /root
COPY . .

ARG PYTORCH_WHL=torch-2.7.1+cpu-cp310-cp310-manylinux_2_28_aarch64.whl
ARG PYTORCH_NPU_WHL=torch_npu-2.7.1.{version}-cp310-cp310-manylinux_2_28_aarch64.whl
ARG APEX_WHL=apex-0.1+ascend-cp310-cp310-linux_aarch64.whl
ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG TOOLKIT_PATH=/usr/local/Ascend/cann
# The CANN version used in the example is 8.5.0. Modify it based on the actual situation during use.
ARG TOOLKIT=Ascend-cann-toolkit_8.5.0_linux-aarch64.run
ARG OPS=Ascend-cann-910b-ops_8.5.0_linux-aarch64.run
ARG TASKD_WHL=taskd-7.3.0-py3-none-linux_aarch64.whl
ARG MINDIO_TTP_WHL=mindio_ttp-1.0.0-py3-none-linux_aarch64.whl
ARG MINDSPEED=MindSpeed
ARG DLLOGGER=dllogger

RUN echo "nameserver 114.114.114.114" > /etc/resolv.conf

RUN echo "deb http://repo.huaweicloud.com/ubuntu-ports/ focal main restricted universe multiverse\n\
deb http://repo.huaweicloud.com/ubuntu-ports/ focal-updates main restricted universe multiverse\n\
deb http://repo.huaweicloud.com/ubuntu-ports/ focal-backports main restricted universe multiverse\n\
deb http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted universe multiverse" > /etc/apt/sources.list

ARG DEBIAN_FRONTEND=noninteractive

# System packages
RUN umask 0022 && apt update && \
    apt-get install -y --no-install-recommends \
    software-properties-common
RUN umask 0022 && add-apt-repository ppa:deadsnakes/ppa && \
    apt update && \
    apt autoremove -y python python3 && \
    apt install -y python3.10 python3.10-dev
# Set up Python symbolic links
RUN ln -s /usr/bin/python3.10 /usr/bin/python
RUN ln -s /usr/bin/python3.10 /usr/bin/python3
RUN ln -s /usr/bin/python3.10-config /usr/bin/python-config
RUN ln -s /usr/bin/python3.10-config /usr/bin/python3-config
# System packages
RUN umask 0022 && apt update && \
        apt-get install -y --no-install-recommends \
        gcc g++ make cmake vim \
        zlib1g zlib1g-dev \
        openssl libsqlite3-dev libssl-dev \
        libffi-dev unzip pciutils \
        net-tools libblas-dev \
        gfortran libblas3 libopenblas-dev \
        curl unzip liblapack3 liblapack-dev \
        libhdf5-dev libxml2 patch
# Time zone
RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime
# Configure pip mirror
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://mirrors.huaweicloud.com/repository/pypi/simple\n\
trusted-host=mirrors.huaweicloud.com' >> ~/.pip/pip.conf
# pip3.10
RUN cd /tmp && \
    apt-get download python3-distutils && \
    dpkg-deb -x python3-distutils_*.deb / && \
    rm python3-distutils_*.deb && \
    cd - && \
    python get-pip.py && \
    rm get-pip.py
RUN umask 0022 && \
    pip install sympy==1.4 && \
    pip install cffi && \
    pip install pathlib2 && \
    pip install grpcio && \
    pip install grpcio-tools && \
    pip install torchvision==0.22.1 && \
    pip install transformers==4.51.0 && \
    pip install absl-py && \
    pip install datasets && \
    pip install tokenizers==0.20.1 && \
    pip install pyOpenSSL
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser
# Install torch, torch_npu, and apex packages
RUN umask 0022 && pip install $PYTORCH_WHL && \
    pip install $PYTORCH_NPU_WHL && \
    pip install $APEX_WHL

# Ascend Packages
# Before building, copy the host's /usr/local/Ascend/driver/version.info to the current directory
RUN umask 0022 &&  \
    cp ascend_install.info /etc/ && \
    mkdir -p /usr/local/Ascend/driver/ && \
    cp version.info /usr/local/Ascend/driver/ && \
    chmod +x $TOOLKIT && \
    chmod +x $OPS

RUN umask 0022 && ./$TOOLKIT --install-path=/usr/local/Ascend/ --install --quiet
RUN echo "source /usr/local/Ascend/cann/set_env.sh" >> ~/.bashrc
RUN umask 0022 && ./$OPS --install --quiet

# After the toolkit package is installed, clear the following files. During container startup, the toolkit package is mounted by Ascend Docker Runtime.
RUN rm -f version.info && rm -f ascend_install.info \
    rm -rf /usr/local/Ascend/driver/

RUN umask 0022 && cd $MINDSPEED && \
    pip install -r requirements.txt && \
    pip install -e . && \
    echo "export PYTHONPATH=/root/MindSpeed:\$PYTHONPATH" >> ~/.bashrc

RUN umask 0022 && cd $DLLOGGER && \
    python setup.py build && \
    python setup.py install

# Import environment variables
ENV HCCL_WHITELIST_DISABLE=1

# Create /lib64/ld-linux-aarch64.so.1
RUN umask 0022 && \
    if [ ! -d "/lib64" ]; \
    then \
        mkdir /lib64 && ln -sf /lib/ld-linux-aarch64.so.1 /lib64/ld-linux-aarch64.so.1; \
    fi

# MindCluster resumable training adaptation script.
RUN umask 0022 && \
    pip install $TASKD_WHL && \
    pip install $MINDIO_TTP_WHL

# Optional. The following commands must be configured when using graceful fault tolerance, pod-level rescheduling, or process-level rescheduling.
RUN sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py

# Install the job scheduling dependency library.
RUN pip install apscheduler

RUN rm -rf tmp && \
    rm -f $PYTORCH_WHL && \
    rm -f $PYTORCH_NPU_WHL && \
    rm -f $APEX_WHL && \
    rm -f $TOOLKIT && \
    rm -f $OPS && \
    rm -f $TASKD_WHL && \
    rm -f $MINDIO_TTP_WHL && \
    rm -rf $DLLOGGER && \
    rm -rf Dockerfile
# Pack the preceding content into the image mindspeed-dl:v1.

If Python 3.10 cannot be installed successfully through PPA, or the deadsnakes PPA does not provide an image source for Python 3.10, you can download the source code and manually compile and install it.

Build the image. Run the following command to generate the image. To make the Dockerfile more secure, you can define a HEALTHCHECK in it based on your service. Check the running status of the container by running the HEALTHCHECK _[OPTIONS]_ CMD command inside the container. Note: Do not omit the the period (".") at the end of the command.

docker build -t mindspeed-dl:v1 .

Building the MindFormers Training Image (MindSpore)

The MindSpore Transformers suite (hereinafter referred to as MindFormers) aims to build a full-process development suite for large model training, fine-tuning, evaluation, inference, and deployment. It provides mainstream Transformer-based pre-trained models and SOTA downstream task applications, covering rich parallel features. It is expected to help users easily implement large model training and innovative research.

The quick start guide in MindSpore Transformers documentation includes installation and quick start chapters, which can be referenced for image creation.

The training image can be built based on the base training image in conjunction with the MindFormers documentation. For details on building the base training image, see the Building a Container Image Using Dockerfile (MindSpore) section.

This section demonstrates how to build a training image based on Ubuntu 20.04, incorporating the steps for building the base training image.

Preparing Software Packages

As shown in Table 1, obtain the software packages for the corresponding operating system, and prepare the Dockerfile and script files required for the image. In the software package names, {version} represents the version number, {arch} represents the architecture, and {chip_type} represents the chip type.

Table 1 Preparing software packages

Software Package	Required	Description	Obtaining Method
MindFormers code repository	Yes	A full-process development suite for building large model training, fine-tuning, evaluation, inference, and deployment, providing mainstream Transformer-based pre-trained models and SOTA downstream applications, covering rich parallel features..	git clone https://gitcode.com/mindspore/mindformers.git cd mindformers git checkout 15ff59dd55b84b4dfc7de03f7f20f6e2be3669ec
requirements.txt file	No	Since dependency installation errors may occur when MindSpore is installed via pip, dependencies can be installed first.	wget https://gitcode.com/mindspore/mindspore/raw/r2.4.1/requirements.txt NOTE The MindSpore package must be used together with Atlas training series products. For details, see the MindSpore Installation Guide.
mindspore-{version}-cp3x-cp3x-linux_aarch64.whl	Yes	MindSpore whl package.. The cp3x in the package name indicates the Python version. For example, x=10 means Python 3.10. Please select the corresponding package based on your actual situation.	Download Link
mindio_ttp-{version}-py3-none-linux_{arch}.whl	Yes	MindIO TFT installation package.	Download Link
Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run	Yes For versions before CANN 8.5.0, the package name is Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run	CANN operator package.	Download Link Note: Please obtain the software package that matches the server.
Ascend-cann-toolkit_{version}_linux-{arch}.run	Yes	CANN Toolkit package.	Download Link Note: Please obtain the software package that matches the server.
taskd-{version}-py3-none-linux_{arch}.whl	Yes	Resumable training whl package.	Download Link Note: In the MindSpore scenario, this whl package must be installed to use graceful fault tolerance, Pod-level rescheduling, process-level rescheduling, and process-level online recovery. This download link points to the TaskD's compressed package Ascend-mindxdl-taskd_{version}_linux-{arch}.zip. You need to decompress it to obtain the corresponding whl package.
version.info	Yes Dependency file for installing CANN.	Driver version information file.	Copy the "/usr/local/Ascend/driver/version.info" file from the host.
ascend_install.info	Yes Dependency file for installing CANN	Driver installation information file.	Copy the "/etc/ascend_install.info" file from the host.
get-pip.py	Yes	Used for installing the pip module.	curl -k https://bootstrap.pypa.io/get-pip.py -o get-pip.py
Dockerfile	Yes	Required for image creation.	-

After downloading the software package, see the OpenPGP Signature Verification Guide to perform PGP digital signature verification on the software package downloaded from the Support website. If the verification fails, do not use the software package and contact Huawei technical support engineers for resolution.

Before installing or upgrading using a software package, you also need to verify the digital signature of the software package following the above process to ensure that the software package has not been tampered with.

For carrier customers, please visit https://support.huawei.com/carrier/digitalSignatureAction.

For enterprise customers, please visit https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054

This section uses a single Atlas 800T A2 training server, Ubuntu 20.04, and Python 3.10 as an example to describe the detailed process of building an image. Modify the relevant steps based on actual conditions during use.

Procedure

Prepare the software packages on the host.

Build the following Dockerfile.

FROM ubuntu:20.04

WORKDIR /root

COPY . .

ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG TOOLKIT_PATH=/usr/local/Ascend/cann
# The CANN version used in this example is 8.5.0. Modify it based on the actual situation during use.
ARG TOOLKIT=Ascend-cann-toolkit_8.5.0_linux-aarch64.run
ARG OPS=Ascend-cann-910b-ops_8.5.0_linux-aarch64.run
ARG MINDIO_TTP_WHL=mindio_ttp-1.0.0-py3-none-linux_aarch64.whl
ARG MINDFORMERS=mindformers
ARG MINDSPORE_REQUIREMENTS=requirements.txt
ARG MINDSPORE_WHL=mindspore-2.5.0-cp310-cp310-linux_aarch64.whl
ARG TASKD_WHL=taskd-7.0.RC1-py3-none-linux_aarch64.whl

RUN echo "nameserver 114.114.114.114" > /etc/resolv.conf

RUN echo "deb http://repo.huaweicloud.com/ubuntu-ports/ focal main restricted universe multiverse\n\
deb http://repo.huaweicloud.com/ubuntu-ports/ focal-updates main restricted universe multiverse\n\
deb http://repo.huaweicloud.com/ubuntu-ports/ focal-backports main restricted universe multiverse\n\
deb http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted universe multiverse" > /etc/apt/sources.list

ARG DEBIAN_FRONTEND=noninteractive

RUN umask 0022 && apt update && \
    apt-get install -y --no-install-recommends \
    software-properties-common
RUN umask 0022 && add-apt-repository ppa:deadsnakes/ppa && \
    apt update && \
    apt autoremove -y python python3 && \
    apt install -y python3.10 python3.10-dev

# Create Python symbolic links
RUN ln -s /usr/bin/python3.10 /usr/bin/python
RUN ln -s /usr/bin/python3.10 /usr/bin/python3
RUN ln -s /usr/bin/python3.10-config /usr/bin/python-config
RUN ln -s /usr/bin/python3.10-config /usr/bin/python3-config

# System packages
RUN umask 0022 && apt update && \
    apt-get install -y --no-install-recommends \
        gcc g++ make cmake vim \
        zlib1g zlib1g-dev \
        openssl libsqlite3-dev libssl-dev \
        libffi-dev unzip pciutils \
        net-tools libblas-dev \
        gfortran libblas3 libopenblas-dev \
        curl unzip liblapack3 liblapack-dev \
        libhdf5-dev libxml2 patch

# Time zone
# RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime

# Configure pip mirror
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://mirrors.huaweicloud.com/repository/pypi/simple\n\
trusted-host=mirrors.huaweicloud.com' >> ~/.pip/pip.conf

# pip3.10
RUN cd /tmp && \
    apt-get download python3-distutils && \
    dpkg-deb -x python3-distutils_*.deb / && \
    rm python3-distutils_*.deb && \
    cd - && \
    python get-pip.py && \
rm get-pip.py

RUN umask 0022 && \
    pip install sympy==1.4 && \
    pip install cffi && \
    pip install pathlib2 && \
    pip install grpcio && \
    pip install grpcio-tools && \
    pip install absl-py && \
    pip install datasets && \
    pip install tokenizers==0.20.1 && \
    pip install pyOpenSSL

# Create the HwHiAiUser and its owner. Ensure the UID and GID are consistent with those on the physical machine to avoid ownerless files. In this example, the user and the corresponding group are automatically created, with both UID and GID set to 1000.
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser

# Ascend packages
# Before building, copy the host's /usr/local/Ascend/driver/version.info to the current directory.
RUN umask 0022 &&  \
    cp ascend_install.info /etc/ && \
    mkdir -p /usr/local/Ascend/driver/ && \
    cp version.info /usr/local/Ascend/driver/ && \
    chmod +x $TOOLKIT && \
    chmod +x $OPS

RUN umask 0022 && ./$TOOLKIT --install-path=/usr/local/Ascend/ --install --quiet
RUN echo "source /usr/local/Ascend/cann/set_env.sh" >> ~/.bashrc
RUN umask 0022 && ./$OPS --install --quiet

# After the toolkit package is installed, clear the following files. During container startup, the toolkit package is mounted by Ascend Docker..
RUN rm -f version.info && \
    rm -rf /usr/local/Ascend/driver/

# Install MindSpore
RUN umask 0022 && pip uninstall te topi hccl -y && \
         pip install sympy && \
         pip install /usr/local/Ascend/cann/lib64/hccl-*-py3-none-any.whl
RUN umask 0022 && \
    pip install -r $MINDSPORE_REQUIREMENTS && \
    pip install $MINDSPORE_WHL

# Install MindFormers
RUN umask 0022 && cd $MINDFORMERS && \
    pip install -r requirements.txt

# MindCluster lossless resumable training script adaptation
RUN umask 0022 && \
    pip install $MINDIO_TTP_WHL --target=$(pip show mindspore | awk '/Location:/ {print $2}') && \
    pip install $TASKD_WHL

# Environment variable
ENV HCCL_WHITELIST_DISABLE=1

# Create /lib64/ld-linux-aarch64.so.1
RUN umask 0022 && \
    if [ ! -d "/lib64" ]; \
    then \
        mkdir /lib64 && ln -sf /lib/ld-linux-aarch64.so.1 /lib64/ld-linux-aarch64.so.1; \
    fi

# Install the job scheduling dependency library
RUN pip install apscheduler

RUN rm -rf tmp && \
    rm -f $TOOLKIT && \
    rm -f $OPS && \
    rm -f $MINDIO_TTP_WHL && \
    rm -f $MINDSPORE_REQUIREMENTS && \
    rm -f $MINDSPORE_WHL
# Package it into the image mindformers-dl:v1

Build the image. Run the following command to generate the image. To make the Dockerfile more secure, you can define a HEALTHCHECK based on your service. Check the running status of the container by running the `HEALTHCHECK_[OPTIONS]_ CMD command inside the container. Note: Do not omit the period (".") at the end of the command.
```
docker build -t mindformers-dl:v1 .
```

Creating a Post-training Image for Reinforcement Learning (Verl)

Verl is a flexible, efficient, and production-ready reinforcement learning training framework designed specifically for the post-training phase of large language models (LLMs).

Building an Image

For details, see Official Verl Documentation - Building an Image. vLLM and Megatron serve as the inference and training backends, respectively.

Install Software

For details, see Official Verl Documentation - Installing Software.

If you need to use the Pod-level rescheduling function, it is recommended that the MindSpeed version is not earlier than the version with commit ID 6390a8ee2f0e59ae237753cce51289a3fe490905.

(Optional) Installing jemalloc

When building an image, you can choose to install jemalloc to optimize memory management. The source code can be downloaded from the jemalloc releases page.

Run the following command to install jemalloc.

tar -xvf jemalloc-{version}.tar.bz2
cd jemalloc-{version}
./configure --prefix=/usr/local
make
make install

After the installation is complete, set the environment variable. Take the installation path /usr/local/lib/libjemalloc.so.2 as an example.
```
export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
```

Script Adaptation

Process Description

Resumable training can be used only after a model script adapts to checkpoints. The general process and logic of script adaptation are shown in Figure 1.

Figure 1 Script adaptation process

Adaptation Example

This sectiond describes adaptation steps for resumable training.

Adaptation Example for PyTorch (MindSpeed-LLM)
Adaptation Example for MindSpor (MindFormers)
Adaptation Example for Reinforcement Learning Post-Training (Verl)

To ensure the normal use of graceful fault tolerance and process-level online recovery, keep clocks of the K8s cluster master node and worker nodes synchronized.
The displayed component code for resumable training is open-source code. For related security instructions, see Security Statement.
The sample code below may differ from the actual implementation. Please use the actual code.
Configure the model parameters according to the settings defined in the model repository. Improper modifications may lead to unexpected issues.
If the error "Failed to bind the IP port. Reason: The IP address and port have been bound already" occurs during training, rectify the fault as follows. For details, see the "HCCL_HOST_SOCKET_PORT_RANGE" section in the CANN Environment Variable Reference.
```
export HCCL_HOST_SOCKET_PORT_RANGE="60000-60050"
export HCCL_NPU_SOCKET_PORT_RANGE="61000-61050"
```
If TaskD is used and the training container uses the host network, first query the current reserved port configuration using sysctl net.ipv4.ip_local_reserved_ports, then add reserved ports 9601 and 9602 using sysctl -w net.ipv4.ip_local_reserved_ports="xxx,9601,9602" (where xxx refers to the previously queried configured ports; omit if none exist).

PAdaptation Example for PyTorch (MindSpeed-LLM)

For training code and dataset preparation, see MindSpeed-LLM User Guide. The following uses two Atlas 800T A2 training servers as an example to describe the specific operation steps.

Pull the training code.

mkdir -p /data/atlas_dls/public/code
cd /data/atlas_dls/public/code
git clone https://gitcode.com/Ascend/MindSpeed-LLM.git
git clone https://github.com/NVIDIA/Megatron-LM.git
cd MindSpeed-LLM
git checkout 2.3.0
cd ..
cd Megatron-LM
git checkout core_v0.12.1
cp -r megatron ../MindSpeed-LLM # Copy the Megatron directory under the Megatron-LM project to the MindSpeed-LLM project.
## Rename MindSpeed-LLM to QWEN3_for_PyTorch_2.7_code.
cd ..
mv MindSpeed-LLM QWEN3_for_PyTorch_2.7_code

Obtain model weights.

Download model weights from Qwen3 and place them in a directory on the server, such as /data/atlas_dls/public/dataset/qwen3-8b-hf.
Prepare a dataset.

Download the detailed dataset, for example, Alpaca, and place it in a directory on the server, such as /data/atlas_dls/public/dataset/qwen3-alpaca.

Process the dataset.

Start the container.

docker run -it -v /data/atlas_dls/public/:/data/atlas_dls/public/ -e ASCEND_VISIBLE_DEVICES=0-7 mindspeed-dl:v1 bash

Perform the following operations in the container.

export TORCH_DEVICE_BACKEND_AUTOLOAD=0
source /usr/local/Ascend/cann/set_env.sh
cd /data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code
# Optional. The following describes how to install the MindSpeed acceleration library. This operation can be performed in any directory. If it has been installed during image creation, skip this operation.
git clone https://gitcode.com/ascend/MindSpeed.git
cd MindSpeed
git checkout 2.3.0_core_r0.12.1
pip install -r requirements.txt
pip install -e .
export PYTHONPATH=/data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/MindSpeed:$PYTHONPATH
cd ..

Process the dataset.

Qwen3 requires that Transformers version be 4.51.0 or later. Therefore, Python 3.9 or later and Transformers 4.51.0 or later must be installed.

python preprocess_data.py \
    --input /data/atlas_dls/public/dataset/qwen3-alpaca/train-00000-of-00001-a09b74b3ef9c3b56.parquet \ # Dataset file path
    --tokenizer-name-or-path /data/atlas_dls/public/dataset/qwen3-8b-hf \ # Open-source model weight file directorye
    --tokenizer-type PretrainedFromHF \
    --handler-name GeneralPretrainHandler \
    --output-prefix /data/atlas_dls/public/dataset/qwen3-alpaca/alpaca \ # Generates alpaca_text_document.bin and .idx files
    --json-keys text \
    --workers 4 \
    --log-interval 1000

If the error "/usr/local/lib/python3.10/dist-packages/sklearn/utils/../../scikit_learn.libs/libgomp-947d5fa1.so.1.0.0: cannot allocate memory in static TLS block" is displayed, run the following command to preload the libgomp library.

export LD_PRELOAD="/usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-947d5fa1.so.1.0.0"

Go to the "mindcluster-deploy" repository, switch to the branch corresponding to the version according to the mindcluster-deploy open-source repository version description, obtain the train_start.sh file from the samples/train/resumable-training/fault-tolerance/without-ranktable/pytorch/Qwen3 directory, and construct the following directory structure on the management node.
```
root@ubuntu:/data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/scripts#
scripts/
└── train_start.sh
```
Obtain the training job YAML. This YAML has already configured pod-level rescheduling, process-level rescheduling, process-level online recovery, elastic training, and more. Set the IP address of the server to which the volume is mounted and rescheduling levels as required.

Training process recovery functions such as process-level rescheduling, process-level online recovery, and elastic training cannot coexist with graceful fault tolerance. For details about how to configure graceful fault tolerance, see Graceful Fault Tolerance Mode.

Configure train_start.sh and the training job YAML based on the actual situation.

Modify the basic parameters of the startup script.

mkdir -p /job/code/alllogs/$MINDX_TASK_ID/ttplogs
mkdir -p /job/code/alllogs/$MINDX_TASK_ID/trainlogs
mkdir -p /job/code/alllogs/$MINDX_TASK_ID/demo/
# Log save path, which can be modified based on the actual situation
export ASCEND_PROCESS_LOG_PATH=/job/code/alllogs/$MINDX_TASK_ID/plogs/$XDL_IP       # Set the plog save path, where $MINDX_TASK_ID is the task UID environment variable injected by Ascend Operator, and $XDL_IP is the environment variable status.hostIP written in the YAML
export TTP_LOG_PATH=/job/code/alllogs/$MINDX_TASK_ID/ttplogs/ttplog$XDL_IP-$RANK    # Set the TTP log save path, where $RANK is the environment variable injected by Ascend Operator for the PyTorch framework
export TRAIN_LOG_PATH=/job/code/alllogs/$MINDX_TASK_ID/trainlogs/$XDL_IP-$RANK      # Set the training log save path
export GLOO_SOCKET_IFNAME=enp189s0f0               # The network port on the physical machine that can communicate. Configure it based on the actual high-speed NIC of the master node. If hostNetwork is set to false in the YAML, set it to eth0.
export HCCL_SOCKET_IFNAME=enp189s0f0               # If hostNetwork is set to false in the YAML, set it to eth0.

CKPT_SAVE_DIR="/job/code/output/ckpt" # The weight save path after training is completed
DATA_PATH="/job/data/alpaca_text_document" # Dataset path. Enter the data path saved during data preprocessing.
TOKENIZER_PATH="/job/data/qwen3-8b-hf" # Tokenizer path. Enter the downloaded open-source weight tokenizer path
CKPT_LOAD_DIR="/job/code/output/ckpt" # Weight loading path

To use TaskD for process-level rescheduling, process-level online recovery, process-level in-place recovery, or elastic training, you also need to start TaskD Manager.

Create a manager.py file in the current directory when calling the training script. The content of the manager.py file is as follows.

from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Number of node
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()

For detailed description of parameters in the manager.py file, see def init_task_manager(config:dict) -> bool:.

Add the following code to the training script to start TaskD Manager.

In the following code, the two statements TASKD_SO_PATH and export LD_PRELOAD are used to configure the path of libtaskd.so after installing TaskD into the environment variable LD_PRELOAD. If these two statements fail to configure successfully, you can manually run the pip show taskd command to obtain the value of Location, concatenate it with /taskd/python/cython_api/libs/libtaskd.so, and then set it via export.

sed -i '/import os/i import taskd.python.adaptor.patch' $(pip3 show torch | grep Location | awk -F ' ' '{print $2}')/torch/distributed/run.py
TASKD_SO_PATH="$(pip show taskd | awk '/^Location: / {print $2"/taskd/python/cython_api/libs/libtaskd.so"}')"
export LD_PRELOAD=$TASKD_SO_PATH:$LD_PRELOAD
export TASKD_PROCESS_ENABLE="on"
if [[ "${RANK}" == 0 ]]; then
    export MASTER_ADDR=${POD_IP}
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &           # The specific execution path of manager.py is determined by the current path, and the error.log path must be created in advance.
fi


torchrun $DISTRIBUTED_ARGS ...

Modify the training job YAML, add a container port, and add port 9601 used for TaskD communication under all Pods (skip if already present).
```
...
        spec:
...
          containers:
...
            ports:
             - containerPort: 9601
               name: taskd-port
...
```

Adaptation Example of MindSpore (MindFormers)

For training code and dataset preparation, see the MindFormers documentation. The following uses two Atlas 900 A3 SuperPoDs as an example to describe the specific steps.

Prepare the code.

mkdir -p /data/atlas_dls/public/code
cd /data/atlas_dls/public/code
git clone https://gitcode.com/mindspore/mindformers.git
cd mindformers
git checkout 15ff59dd55b84b4dfc7de03f7f20f6e2be3669ec
# Rename mindformers to QWEN3_for_MS_code
cd ..
mv mindformers QWEN3_for_MS_code

Prepare a dataset.

Download DagsHub and place it in a directory on the server, such as /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset.

Convert the dataset.

Download the dataset conversion script.

Download the dataset conversion script from Dataset Conversion and place it in a directory on the server, such as /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/gen_wiki_json.py.
Download the tokenizer file.

Download the tokenizer file from Qwen3-32B and place it in a directory on the server, such as /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/Qwen3-32B-tokenizer.

Convert the dataset.

Start the container and mount the required files.

docker run -it -v /data/atlas_dls/public/code/:/data/atlas_dls/public/code/ mindformers-dl:v1 bash

Execute the conversion script to convert wiki.train.tokens to jsonl format.

# Prepare the Python environment required to run this script in advance.
cd /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset
python gen_wiki_json.py --input wiki.train.tokens  --output wiki.jsonl

Convert the data from the jsonl format to the bin format.

# If the error "ModuleNotFoundError: No module named 'xxx'" is reported during execution, install the dependency yourself.
cd /data/atlas_dls/public/code/QWEN3_for_MS_code
python toolkit/data_preprocess/megatron/preprocess_indexed_dataset.py \
  --input /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/wiki.jsonl \
  --output-prefix /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/wiki103-megatron \
  --tokenizer-type HuggingFaceTokenizer \
  --tokenizer-dir /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/Qwen3-32B-tokenizer # For models of other specifications, you can adjust to the corresponding tokenizer path.

After the execution is complete, the files wiki103-megatron_text_document.bin and wiki103-megatron_text_document.idx will be generated in the /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset directory. When specifying the dataset path, use /data/atlas_dls/public/code/QWEN3_for_MS_code/dataset/wiki103-megatron_text_document without the file extension.

Obtain the training job YAML and the training launch script, and modify them.
1. If the value of the hostNetwork parameter in the training job YAML is false, you need to set the value of GLOO_SOCKET_IFNAME in the startup script to eth0. An example is as follows:
```
export GLOO_SOCKET_IFNAME=eth0  #eth0 is the network interface that can communicate within the container.
export HCCL_SOCKET_IFNAME=eth0
```
  Then modify other parameters in the startup script based on the actual situation.
2. Modify configurations such as the server IP address of the mounted volume in the job YAML based on actual conditions.
3. To use TaskD for process-level rescheduling, process-level online recovery, process-level in-place recovery, suspension and switchback of cross-rail communication tasks, or online stress testing, you also need to start TaskD Manager.
  1. Create a manager.py file and place it in the current directory when calling the training script. The content of the manager.py file is as follows.
```
from taskd.api import init_taskd_manager, start_taskd_manager
import os

job_id=os.getenv("MINDX_TASK_ID")
node_nums=XX         # Total number of nodes
proc_per_node=XX     # Number of training processes per node

init_taskd_manager({"job_id":job_id, "node_nums": node_nums, "proc_per_node": proc_per_node})
start_taskd_manager()
```
    NOTE For details about the parameters in the manager.py file, see def init_taskd_manager(config:dict) -> bool:.
  2. Add the following code to the training script to start TaskD Manager. In the code, the first two statements are used to configure the path of libtaskd.so to the environment variable LD_PRELOAD after TaskD is installed. If the two statements fail to be configured, run the pip show taskd command to obtain the value of Location, combine the value with /taskd/python/cython_api/libs/libtaskd.so, and run the export command.
```
TASKD_SO_PATH="$(pip show taskd | awk '/^Location: / {print $2"/taskd/python/cython_api/libs/libtaskd.so"}')"
export LD_PRELOAD=$TASKD_SO_PATH:$LD_PRELOAD
export TASKD_PROCESS_ENABLE="on"
if [[ "${MS_SCHED_HOST}" == "${POD_IP}" ]]; then
    python /job/code/manager.py 2>> /job/code/alllogs/$MINDX_TASK_ID/taskd/error.log &   # The specific execution path of manager.py is determined by the current path. The error.log path must be created in advance.
fi
msrun ...
```
  3. Modify the training job YAML by adding port 9601 for TaskD communication under all Pods (skip if it already exists).
```
...
        spec:
...
          containers:
...
            ports:
             - containerPort: 9601
               name: taskd-port
...
```

Modify the parameter model configuration file.

Open the configs/qwen3/pretrain_qwen3_32b_4k.yaml file in the code directory.
```
vi configs/qwen3/pretrain_qwen3_32b_4k.yaml
```

Press i to enter insert mode and modify the parameter model configuration file.

Modify the following configurations in bold, including the dataset path, distributed parallel parameters, and model parameters. The following model parameters are for reference only. Modify them as required.

 train_dataset: &train_dataset
   data_loader:
     type: BlendedMegatronDatasetDataLoader
     datasets_type: "GPTDataset"
     sizes:
       - 8000  # Number of samples in the training dataset
       - 0     # Number of samples in the test dataset (currently unsupported)
       - 0     # Number of samples in the evaluation dataset (currently unsupported)
     config:
       seed: 1234  # Random seed for data sampling
       split: "1, 0, 0"  # Proportions for training, test, and evaluation datasets (test/eval currently unsupported)
       seq_length: 4096  # Sequence length of the dataset
       eod_mask_loss: False  # Whether to calculate loss at the end-of-document (EOD)
       reset_position_ids: False  # Whether to reset position_ids at EOD
       create_attention_mask: True  # Whether to include attention_mask in the dataset
       reset_attention_mask: False  # Whether to reset attention_mask at EOD, creating a stepped attention_mask
       create_compressed_eod_mask: False  # Whether to include a compressed attention_mask
       eod_pad_length: 128  # Length of the compressed attention_mask
       eod: 1  # Token ID for EOD in the dataset
       pad: -1  # Token ID for padding in the dataset
       data_path:  # Sampling proportion and path for the Megatron dataset
         - '1'
         - "/job/data/wiki103-megatron_text_document" # Dataset path
 ……
 # Parallel configuration
 parallel_config:
   data_parallel: &dp 4  # Number of data parallel. If using the high availability feature, it must be an even number.
   model_parallel: 8  # Number of model parallel
   pipeline_stage: 1  # Number of pipeline parallel
   micro_batch_num: 1  # Pipeline parallel microbatch size
   use_seq_parallel: False  # Whether to enable sequence parallelism
   gradient_aggregation_group: 1  # Size of the gradient communication operator fusion group
 # When model_parallel > 1, setting micro_batch_interleave_num to 2 may accelerate the training process.
 micro_batch_interleave_num: 1
 ……
 model:
   model_config:
 # Configurations from Hugging Face
     vocab_size: 75968            # Model parameters reduced here for testing purposes only. Adjust as needed.
     hidden_size: 2560           # Model parameters reduced here for testing purposes only. Adjust as needed.
     intermediate_size: 12800   # Model parameters reduced here for testing purposes only. Adjust as needed.
     num_hidden_layers: 32      # Model parameters reduced here for testing purposes only. Adjust as needed.
     num_attention_heads: 32    # Model parameters reduced here for testing purposes only. Adjust as needed.
     num_key_value_heads: 8
     head_dim: 128
     hidden_act: 'swiglu'
     max_position_embeddings: 4096
     seq_length: 4096
     initializer_range: 0.02
     rms_norm_eps: 1.e-6
     use_cache: True
     tie_word_embeddings: False
     rope_theta: 1000000.
     attention_bias: False
     use_flash_attention: True
     add_bias_linear: False
     eos_token_id: 151645
     pad_token_id: 151643
     bos_token_id: 151643
     attention_dropout: 0.0
 # Configurations from MindFormers
     hidden_dropout: 0.0
     input_sliced_sig: True
     untie_embeddings_and_output_weights: True
     position_embedding_type: "rope"
     qk_layernorm: True
     use_contiguous_weight_layout_attention: False
     qkv_concat: True
     offset: [0]
     params_dtype: "float32"
     compute_dtype: "bfloat16"
     layernorm_compute_dtype: "float32"
     softmax_compute_dtype: "float32"
     rotary_dtype: "float32"
     residual_dtype: "float32"
     model_type: "qwen3"
     architectures: ["Qwen3ForCausalLM"]

(Optional) If dying gasp checkpoint is used, and the checkpoint is loaded through Pod-level rescheduling after being saved, modify the following fields.

For the first startup, ensure that the directory specified by the load_checkpoint parameter contains a valid checkpoint or is empty; otherwise, the training may fail to start properly.
```
resume_training: True
src_strategy_path_or_dir: './output/strategy'
load_checkpoint: './output/checkpoint'
```

Press Esc, type :wq!, and press Enter to save and exit editing.

Adaptation Example of Reinforcement Learning Post-Training (Verl)

MindCluster only supports Job-level and Pod-level rescheduling for the Verl framework, where Pod-level rescheduling only supports the GRPO algorithm. Verl's training jobs are managed by a Ray cluster. To adapt to MindCluster's Ascend Job deployment, one Pod is deployed on each Worker node, and the Pod hosts all processes on that Ray cluster. The head node of the Ray cluster is the Master Pod. The head node starts the Ray cluster, and other Worker Pods join the Ray cluster after starting, after which the head node submits jobs.

The following uses two Atlas 900 A3 SuperPoDs as an example to describe the specific operation steps.

Prepare the Verl code.

git clone https://github.com/volcengine/verl.git
cd verl
git checkout b97ebfd5062223337ae065c2250f8ab5c0e08e5e
rm -rf recipe
git clone https://github.com/verl-project/verl-recipe.git
cd verl-recipe
git checkout 474494acafc6482a7e16be2c82e957bd8ca11a3f
cd ..
mv verl-recipe recipe

Download Qwen/Qwen3-32B.

Obtain and convert the openai/gsm8k dataset.

Prepare the dataset directory.

Create a gsm8k/main directory under a certain directory (for example, /data/dataset/). Place the downloaded dataset files train-00000-of-00001.parquet and test-00000-of-00001.parquet into the /data/dataset/gsm8k/main directory.
Start the container.

The following example assumes that both the Verl code and dataset files are located in the /data path. Modify the mount path according to your actual situation.
```
docker run -it \
-v /data:/data \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
verl:v1 /bin/bash
```

Go to the code directory and modify the gsm8k.py script.

cd /data/code/verl
vi examples/data_preprocess/gsm8k.py

Locate the following code snippet:

if local_dataset_path is not None:
    dataset = datasets.load_dataset(local_dataset_path, "main")
else:
    dataset = datasets.load_dataset(data_source, "main")

Modify it to:

if local_dataset_path is not None:
    dataset = datasets.load_dataset(local_dataset_path)
else:
    dataset = datasets.load_dataset(data_source, "main")

Perform preprocessing.

 python3 examples/data_preprocess/gsm8k.py \
 --local_save_dir /data/datasets/gsm8k \
 --local_dataset_path /data/datasets/gsm8k

Prepare the training script and configuration files.

Obtain the training script and configuration files.

Obtain the sample training scripts start_grpo.sh and run_grpo_qwen3_32b_a3b_megatron.sh from the sample repository and place them in the Verl root directory. Obtain the configuration file runtime_env.yaml and place it in the verl/recipe/fault_recover/config directory.

Modify the training scripts.

Modify start_grpo.sh:

# NIC configuration (modify based on actual NIC information when hostNetwork=true)
export HCCL_SOCKET_IFNAME=eth0
export TP_SOCKET_IFNAME=eth0
export GLOO_SOCKET_IFNAME=eth0

# Dependency path (configure based on actual path)
export PYTHONPATH=$PYTHONPATH:/data/code/Megatron-LM

# Log directory (configure based on actual path)
export path_log_dir=/data/logs/$MINDX_TASK_ID/trainlog
export ASCEND_PROCESS_LOG_PATH=/data/logs/$MINDX_TASK_ID/plog

# Memory optimization library (configure based on actual path)
export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2:$LD_PRELOAD

Modify run_grpo_qwen3_32b_a3b_megatron.sh:

  MODEL_PATH=/data/models/Qwen3-32B                 # Model path. Modify it based on the actual situation.
  CKPTS_DIR="/data/ckpt/Qwen3-32B-save/"            # Path for saving checkpoints. Modify it based on the actual situation. Shared storage is recommended.
  TRAIN_FILE="/data/datasets/gsm8k/train.parquet"   # Dataset path. Modify it based on the actual situation.
  TEST_FILE="/data/datasets/gsm8k/test.parquet"     # Dataset path. Modify it based on the actual situation.

Prepare the job YAML and submit the job.

Obtain verl-grpo.yaml (the example file configures Pod-level and Job-level rescheduling by default, which can be modified based on actual conditions), and run the following startup command:

```shell
kubectl apply -f verl-grpo.yaml
```

After startup, error messages similar to the following may appear in the logs. This is normal because the head node usually does not have an NPU mounted.

```ColdFusion
[ERROR] RUNTIME(38734,python3): ... [driver.cc:64]38734 GetDeviceCount:Call drvGetDevNum, drvRetCode=7.
[ERROR] ASCENDCL(38734,python3): ... aclrtGetDeviceCountImpl:get device count failed, runtime result = 507899.
[ERROR] APP(38734,python3): ... "[PTA]:"get device count of NPU failed""
```

Preparation of Job YAML Files

The cluster scheduling components provide YAML examples. You need to select an appropriate YAML example based on the functionality, model type, job type, and fault handling mode, and make necessary modifications according to actual requirements before using it.

Table 1 Training job YAML examples

Job Type	Hardware Model	Training Framework	Model	YAML File Name	Link	Description
Ascend Job	Atlas 800T A2 training server Atlas 900 A2 PoD cluster basic unit	PyTorch	Qwen3	pytorch_multinodes_acjob_910b.yaml	pytorch_multinodes_acjob_910b.yaml	A two-server eight-processor job is presented in the example file by default.
Ascend Job	Atlas 800T A2 training server Atlas 900 A2 PoD cluster basic unit	MindSpore	Qwen3	ms_multinodes_acjob_superpod.yaml	ms_multinodes_acjob_superpod.yaml	A two-server 16-processor job is presented in the example file by default.
Ascend Job	Atlas 900 A3 SuperPoD	Verl	Qwen3-30B	verl-resche.yaml	verl-resche.yaml	A two-server 16-processor job is presented in the example file by default.

Currently, resumable training does not provide an example YAML for the Atlas 900 A3 SuperPoD. You can add the annotations field under labels in the example YAML. An example is as follows:

...
 labels:
...
 annotations:
   sp-block: "32"   # Number of processors on a logical SuperPoD. For details about the sp-block field, see YAML Parameters.
...

Submitting a Job

In the example YAML file, the job is deployed in the default namespace. This section uses the PyTorch framework as an example to describe how to submit a training job.

On the management node, run the following command to submit the training job using the YAML file.

kubectl apply -f XXX.yaml

Example:

kubectl apply -f pytorch_multinodes_acjob_910b.yaml

The following information is displayed:

configmap/reset-config-default-test-pytorch created
ascendjob.mindxdl.gitee.com/default-test-pytorch created

Viewing Job Progress

After the training job is successfully submitted, it runs normally. You can view the running status of the training job as follows.

Viewing All Training Jobs

To view all training jobs running on the current node, follow the steps below.

Run the following command to view the running status of training jobs.

kubectl get pods -A -o wide

Command output:

NAMESPACE        NAME                                       READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES
default          default-test-pytorch-master-0              1/1     Running   0          5s    xxx.xxx.xxx.xxx   node1          <none>           <none>
default          default-test-pytorch-worker-0              1/1     Running   0          5s    xxx.xxx.xxx.xxx   node2          <none>           <none>
……

Viewing the Training Job of a Single Pod

View the training job running on one of the Pod. The procedure is as follows.

Run the following command to view the running status of the training job.

kubectl logs default-test-pytorch-worker-0 -n default -f

The following is an example of the command output. The presence of "loss" indicates that the job is running normally.

Checking Whether Checkpoint Files Exist

The fault recovery function is implemented based on checkpoint files. You need to check whether the checkpoint files exist on the storage node.

You can wait until the training job has been running longer than the user-configured interval for saving checkpoint files, then check whether periodic checkpoint files exist in the configured checkpoint file save path. The operation steps are as follows.

Log in to the storage node and run the following command to enter the checkpoint file path.
```
cd /data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/output/ckpt
```
Run the following command to check whether periodic checkpoint files exist in the current directory.
```
ll ./
```
The following is an example of the command output, indicating that periodic checkpoint files exist.
```
total 8
drwx-xr-x-  18 root root   8192 Jun 22 18:39 iter_0000100
-rw-r--r--  1 root root    2    Jun 22 18:39 latest_checkpointed_iteration.txt
```
(Optional) If dying gasp is used, you can run the following command in the path where checkpoint files are saved to check whether dying gasp checkpoint files exist in the current directory.
```
ll ./
```
The following is an example of the command output, indicating that dying gasp files exist.
```
total 8
drwx-xr-x-  18 root root   8192 Jun 22 15:39 iter_0000009
-rw-r--r--  1 root root    2    Jun 22 15:39 latest_checkpointed_iteration.txt
```

Viewing Training Results

(Optional) Constructing Faults

This section describes how to construct simple faults, including node faults, parameter plane network faults, and service plane faults.

Constructing chip faults poses security risks. If needed, contact Huawei technical support engineers for assistance.

Constructing a Node Fault

Restart the training node to simulate node status loss caused by node power-off. This fault can be automatically rectified after the node is restarted.

After an iteration ends in a normal training job, log in to the node that is being trained.
Run the following commandto restart the training node to simulate a node status loss fault.
```
reboot
```
Run the following command multiple times on the master node to check the pod status.
```
kubectl get pod -A
```
You can see that the pod status changes from Terminating to Pending, and finally to Running, indicating that the training job has been restarted.

Run the following command on the master node to view the training logs and record the time when resumable training succeeds.

kubectl logs -n Namespace_name Pod_name

The sample output below shows that the latest checkpoint file from the ninth iteration is used to resume training at the tenth iteration after a fault occurs.

[2025-06-22 14:47:00] iteration       10/    5000 | consumed samples:          640 | elapsed time per iteration (ms): 1932.5 | learning rate: 2.500000E-07 | global batch size:    64 | lm loss: 1.053084E+01 | loss scale: 1.0 | g      rad norm: 56.739 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
[2025-06-22 14:47:02] iteration       11/    5000 | consumed samples:          704 | elapsed time per iteration (ms): 1981.0 | learning rate: 2.750000E-07 | global batch size:    64 | lm loss: 1.044677E+01 | loss scale: 1.0 | g      rad norm: 57.590 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
......

Constructing a Parameter Plane Network Fault

A parameter plane network fault can be simulated by disconnecting the NPU network link. NPU network faults do not affect single-server training jobs. After the link is disconnected, you need to manually restore the link. Otherwise, the fault persists.

After an iteration ends in a normal training job, log in to the node that is being trained.
Run the following command to simulate an NPU network link fault.
```
hccn_tool -i {device_id} -link -s down
```
device_id is the NPU ID, which can be viewed using the npu-smi info command.
Run the following command to check the NPU link status.
```
hccn_tool -i {device_id} -net_health -g
```
If the following information is displayed, the NPU network link fault is successfully created.
```
net health status: Fault
```
Run the following command multiple times on the master node to check the pod status.
```
kubectl get pod -A
```
The pod status changes from Terminating to Pending and finally to Running, indicating that the training job has been restarted.

Run the following command on the master node to view the training logs and record the time when the training is successfully resumed.

kubectl logs -n Namespace_name Pod_name

The sample output below shows that the latest checkpoint file from the ninth iteration is used to resume training at the tenth iteration after a fault occurs.

[2025-06-22 14:47:00] iteration       10/    5000 | consumed samples:          640 | elapsed time per iteration (ms): 1932.5 | learning rate: 2.500000E-07 | global batch size:    64 | lm loss: 1.053084E+01 | loss scale: 1.0 | g      rad norm: 56.739 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
[2025-06-22 14:47:02] iteration       11/    5000 | consumed samples:          704 | elapsed time per iteration (ms): 1981.0 | learning rate: 2.750000E-07 | global batch size:    64 | lm loss: 1.044677E+01 | loss scale: 1.0 | g      rad norm: 57.590 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
......

Run the following command to recover from the NPU network link fault.
```
hccn_tool -i {device_id} -cfg recovery
```
Run the following command to check the NPU link status.
```
hccn_tool -i {device_id} -net_health -g
```
The following is an example of the echo output, indicating that the NPU network link fault has been recovered.
```
net health status: Success
```

Constructing a Service Plane Fault

A service plane fault can be simulated by deleting the training process.

After an iteration ends in a normal training job, log in to the node that is being trained.
Run the following command to query training process information using the training startup script.
```
ps -ef | grep python| grep Training startup script.py
```
Run the following command to manually delete the training process with the smallest PID.
```
kill -9 pid
```
Run the following command multiple times on the master node to check the pod status.
```
kubectl get pod -A
```
The pod status changes from Terminating to Pending and finally to Running, indicating that the training job has been restarted.

Run the following command on the master node to view the training logs and record the time when resumable training succeeds.

kubectl logs -n Namespace_name Pod_name

The sample output below shows that the latest checkpoint file from the ninth iteration is used to resume training at the tenth iteration after a fault occurs.

[2025-06-22 14:47:00] iteration       10/    5000 | consumed samples:          640 | elapsed time per iteration (ms): 1932.5 | learning rate: 2.500000E-07 | global batch size:    64 | lm loss: 1.053084E+01 | loss scale: 1.0 | g      rad norm: 56.739 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
[2025-10-16 14:47:02] iteration       11/    5000 | consumed samples:          704 | elapsed time per iteration (ms): 1981.0 | learning rate: 2.750000E-07 | global batch size:    64 | lm loss: 1.044677E+01 | loss scale: 1.0 | g      rad norm: 57.590 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
......

Rescheduling Mode

Rescheduling Scenarios

When a node is faulty, Volcano schedules its training jobs to other nodes that meet the requirements.

kubectl get pods -A -o wide

Before the fault occurs, assume that the training jobs are scheduled to node1 and node2. After node1 is faulty, Volcano reschedules the training jobs on node1 and node2 to node2 and node3. The following shows a sample command output after rescheduling:

NAMESPACE        NAME                                       READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES
default          default-test-pytorch-master-0              1/1     Running   0          5s    xxx.xxx.xxx.xxx   node2          <none>           <none>
default          default-test-pytorch-worker-0              1/1     Running   0          5s    xxx.xxx.xxx.xxx   node3          <none>           <none>
……

Checking the Job Running Status on a Single Pod

Run the following command to check the running status of the training job on a single pod:

kubectl logs default-test-pytorch-worker-0 -n default -f

The sample output below shows that the latest checkpoint file from the ninth iteration is used to resume training at the tenth iteration after a fault occurs.

2025-09-08 11:34:00.400331 warn 1900637 [77840][PYH tft_replica_optimizer.py:659] Replica optimizer increase Memory On Chip Usage by:0.6572 GB!
2025-09-08 11:34:00.401841 warn 1900631 [28432][PYH tft_replica_optimizer.py:659] Replica optimizer increase Memory On Chip Usage by:0.6572 GB!
2025-09-08 11:34:00.402489 warn 1900639 [10928][PYH tft_replica_optimizer.py:659] Replica optimizer increase Memory On Chip Usage by:0.6572 GB!
2025-09-08 11:34:00.426989 warn 1900627 [98608][PYH tft_replica_optimizer.py:659] Replica optimizer increase Memory On Chip Usage by:0.6572 GB!
2025-09-08 11:34:00.429141 warn 1900634 [24592][PYH tft_replica_optimizer.py:659] Replica optimizer increase Memory On Chip Usage by:0.6572 GB!
(min, max) time across ranks (ms):
    load-checkpoint ................................: (32107.12, 32108.53)
(min, max) time across ranks (ms):
    model-and-optimizer-setup ......................: (32528.79, 32544.35)
    train/valid/test-data-iterators-setup ..........: (72.68, 656.79)
[rank16]:[W908 11:34:01.252908110 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
[rank24]:[W908 11:34:01.254614170 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
[rank17]:[W908 11:34:01.421349990 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
[rank20]:[W908 11:34:01.431165020 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
[rank19]:[W908 11:34:01.431240250 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
[rank30]:[W908 11:34:01.431707980 compiler_depend.ts:335] Warning: Cannot create tensor with interal format while allow_internel_format=False, tensor will be created with base format. (function operator())
...
/root/MindSpeed/mindspeed/core/fp8_utils.py:11: UserWarning: Currently, it is not supported to Cast shard fp32 main params to fp8 model params
  warnings.warn("Currently, it is not supported to Cast shard fp32 main params to fp8 model params")
/root/MindSpeed/mindspeed/core/fp8_utils.py:11: UserWarning: Currently, it is not supported to Cast shard fp32 main params to fp8 model params
  warnings.warn("Currently, it is not supported to Cast shard fp32 main params to fp8 model params")
 [2025-09-08 11:37:00] iteration       10/    5000 | consumed samples:          640 | elapsed time per iteration (ms): 6932.5 | learning rate: 2.500000E-07 | global batch size:    64 | lm loss: 1.053084E+01 | loss scale: 1.0 | g      rad norm: 56.739 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 [2025-09-08 11:37:03] iteration       11/    5000 | consumed samples:          704 | elapsed time per iteration (ms): 1981.0 | learning rate: 2.750000E-07 | global batch size:    64 | lm loss: 1.044677E+01 | loss scale: 1.0 | g      rad norm: 57.590 | num zeros: 0 | number of skipped iterations:   0 | number of nan iterations:   0 |
...

Viewing Job Rescheduling Records

Run the following command to view job rescheduling records.

kubectl describe cm -n mindx-dl job-reschedule-reason

Command output:

Name:         job-reschedule-reason
Namespace:    mindx-dl
Labels:       <none>
Annotations:  <none>
Data
====
recent-reschedule-records:
----
{"default/default-test-pytorch-141274b7-ce93-4d31-adde-6c24456a8a3b":{"JobID":"default/default-test-pytorch-141274b7-ce93-4d31-adde-6c24456a8a3b","TotalRescheduleTimes":1,"RescheduleRecords":[{"LogFileFormatTime":"I0908 11:36:10","RescheduleTimeStamp":1759683370,"ReasonOfTask":[{"RescheduleReason":"pod-failed","PodName":"default-test-pytorch-worker-0","NodeName":"node2","NodeRankIndex":"1"}]}]}}
Events:  <none>

Graceful Fault Tolerance Mode

This section describes how to view the information about a training process that uses the graceful fault tolerance mode for fault handling. When a processor fault occurs, graceful fault tolerance processing is carried out after the process exits. Once the fault has been rectified, the process restarts.

Log Description

Training logs of the restarted training process are stored in training script path_newlog, with details as follows.

QWEN3 (PyTorch) training logs: /data/atlas_dls/public/code/QWEN3_for_PyTorch_2.7_code/alllogs
QWEN3 (MindSpore) training logs: /data/atlas_dls/public/code/QWEN3_for_MS_code/alllogs

Procedure

Log in to the management node and run the following command to check the chip status.
```
npu-smi info
```
If the following information is displayed, the training process occupies the on-chip memory and runs properly.
After a fault occurs, run the following command to check the chip information.
```
npu-smi info
```
If the following information is displayed, the training process has exited and the on-chip memory has been freed.
After fault recovery, run the following command to view chip information.
```
npu-smi info
```
If the following information is displayed, the training process has been restarted. It now occupies the on-chip memory and runs properly.

Deleting a Job

Procedure

Run the following command in the YAML directory of a delivered training job to delete the job:

kubectl delete -f XXX.yaml

Example:

kubectl delete -f pytorch_multinodes_acjob_910b.yaml

Command output:

configmap "reset-config-default-test-pytorch" deleted
ascendjob.mindxdl.gitee.com "default-test-pytorch" deleted

Operations and Maintenance

Prerequisites

This function is only applicable to specific scenarios where you need to use the rescheduling function and the autoStowing parameter has been set to false in the startup YAML of Ascend Device Plugin.

Procedure

Run the following command to add the chips whose health status is restored from unhealthy to healthy to the resource pool.
```
kubectl label nodes node_name huawei.com/Ascend910-Recover-
```
After executing this command, the huawei.com/Ascend910-Recover label will be deleted, and the chips with this label will be returned to the resource pool for scheduling.

NOTE This command is only used to clear the Recover label information. Do not use it to add labels.
Run the following command to return chips whose parameter plane network health status has recovered from unhealthy to healthy to the resource pool.
```
kubectl label nodes node_name huawei.com/Ascend910-NetworkRecover-
```
After executing this command, the huawei.com/Ascend910-NetworkRecover label will be deleted, and the corresponding chips in huawei.com/Ascend910-NetworkUnhealthy will also be cleared.

NOTE This command is only used to clear the NetworkRecover label information. Do not use it to add labels.