AscendJob

AscendJob (acjob): a job type customized by MindCluster. You can start a training or inference job of this type by configuring resource information using environment variables or files.

Supported Framework

MindSpore
TensorFlow
PyTorch

Example

pytorch_multinodes_acjob_910b.yaml:

apiVersion: mindxdl.gitee.com/v1
kind: AscendJob
metadata:
  name: default-test-pytorch
  labels:
    framework: pytorch
    ring-controller.atlas: ascend-910b
    tor-affinity: "null" # This label determines whether a job uses the switch affinity scheduling feature. If the value is null or the label is not specified, this feature is not used. large-model-schema indicates a foundation model job or padding job, and normal-schema indicates a common job.
spec:
  schedulerName: volcano   # work when enableGangScheduling is true
  runPolicy:
    schedulingPolicy:      # work when enableGangScheduling is true
      minAvailable: 2
      queue: default
  successPolicy: AllWorkers
  replicaSpecs:
    Master:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            ring-controller.atlas: ascend-910b
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: job-name
                        operator: In
                        values:
                          - default-test-pytorch
                  topologyKey: kubernetes.io/hostname
          nodeSelector:
            host-arch: huawei-arm
            accelerator-type: card-910b-2 # depend on your device model, 910bx8 is module-910b-8 ,910bx16 is module-910b-16
          containers:
          - name: ascend # do not modify
            image: pytorch-test:latest         # training framework image, which can be modified
            imagePullPolicy: IfNotPresent
            env:
              - name: XDL_IP                                       # IP address of the physical node, which is used to identify the node where the pod is running
                valueFrom:
                  fieldRef:
                    fieldPath: status.hostIP
              # ASCEND_VISIBLE_DEVICES env variable is used by ascend-docker-runtime when in the whole card scheduling scene with volcano scheduler. 
              # Please delete it when in the static vNPU scheduling, dynamic vNPU scheduling, volcano without Ascend-volcano-plugin, without volcano scenes.
              - name: ASCEND_VISIBLE_DEVICES
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.annotations['huawei.com/Ascend910']               # The value must be the same as resources.requests
            command:                           # training command, which can be modified
              - /bin/bash
              - -c
            args: [ "cd /job/code/scripts; chmod +x train_start.sh; bash train_start.sh /job/code /job/output main.py --data=/job/data/resnet50/imagenet --amp --arch=resnet50 --seed=49 -j=128 --world-size=1 --lr=1.6 --dist-backend='hccl' --multiprocessing-distributed --epochs=90 --batch-size=4096" ]
            ports:                          # default value containerPort: 2222 name: ascendjob-port if not set
              - containerPort: 2222         # determined by user
                name: ascendjob-port        # do not modify
            resources:
              limits:
                huawei.com/Ascend910: 2
              requests:
                huawei.com/Ascend910: 2
            volumeMounts:
            - name: code
              mountPath: /job/code
            - name: data
              mountPath: /job/data
            - name: output
              mountPath: /job/output
            - name: ascend-driver
              mountPath: /usr/local/Ascend/driver
            - name: ascend-add-ons
              mountPath: /usr/local/Ascend/add-ons
            - name: dshm
              mountPath: /dev/shm
            - name: localtime
              mountPath: /etc/localtime
          volumes:
          - name: code
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/public/code/ResNet50_ID4149_for_PyTorch/"
          - name: data
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/public/dataset/"
          - name: output
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/output/"
          - name: ascend-driver
            hostPath:
              path: /usr/local/Ascend/driver
          - name: ascend-add-ons
            hostPath:
              path: /usr/local/Ascend/add-ons
          - name: dshm
            emptyDir:
              medium: Memory
          - name: localtime
            hostPath:
              path: /etc/localtime
    Worker:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            ring-controller.atlas: ascend-910b
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: job-name
                        operator: In
                        values:
                          - default-test-pytorch
                  topologyKey: kubernetes.io/hostname
          nodeSelector:
            host-arch: huawei-arm
            accelerator-type: card-910b-2 # depend on your device model, 910bx8 is module-910b-8 ,910bx16 is module-910b-16
          containers:
          - name: ascend # do not modify
            image: pytorch-test:latest                # Training framework image, which can be modified
            imagePullPolicy: IfNotPresent
            env:
              - name: XDL_IP                                       # IP address of the physical node, which is used to identify the node where the pod is running
                valueFrom:
                  fieldRef:
                    fieldPath: status.hostIP
              # ASCEND_VISIBLE_DEVICES env variable is used by ascend-docker-runtime when in the whole card scheduling scene with volcano scheduler. 
          # Please delete it when in the static vNPU scheduling, dynamic vNPU scheduling, volcano without Ascend-volcano-plugin, without volcano scenes.
              - name: ASCEND_VISIBLE_DEVICES
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.annotations['huawei.com/Ascend910']               # The value must be the same as resources.requests
            command:                                  # training command, which can be modified
              - /bin/bash
              - -c
            args: ["cd /job/code/scripts; chmod +x train_start.sh; bash train_start.sh /job/code /job/output main.py --data=/job/data/resnet50/imagenet --amp --arch=resnet50 --seed=49 -j=128 --world-size=1 --lr=1.6 --dist-backend='hccl' --multiprocessing-distributed --epochs=90 --batch-size=4096"]
            ports:                          # default value containerPort: 2222 name: ascendjob-port if not set
              - containerPort: 2222         # determined by user
                name: ascendjob-port        # do not modify
            resources:
              limits:
                huawei.com/Ascend910: 2
              requests:
                huawei.com/Ascend910: 2
            volumeMounts:
            - name: code
              mountPath: /job/code
            - name: data
              mountPath: /job/data
            - name: output
              mountPath: /job/output
            - name: ascend-driver
              mountPath: /usr/local/Ascend/driver
            - name: ascend-add-ons
              mountPath: /usr/local/Ascend/add-ons
            - name: dshm
              mountPath: /dev/shm
            - name: localtime
              mountPath: /etc/localtime
          volumes:
          - name: code
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/public/code/ResNet50_ID4149_for_PyTorch/"
          - name: data
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/public/dataset/"
          - name: output
            nfs:
              server: 127.0.0.1
              path: "/data/atlas_dls/output/"
          - name: ascend-driver
            hostPath:
              path: /usr/local/Ascend/driver
          - name: ascend-add-ons
            hostPath:
              path: /usr/local/Ascend/add-ons
          - name: dshm
            emptyDir:
              medium: Memory
          - name: localtime
            hostPath:
              path: /etc/localtime

Key Fields

**Table 1** acjob fields
Field Path	Type	Format	Description
apiVersion	String	-	Object's versioning resource pattern. The server will convert it to the latest internal value and reject unrecognized versions. For more information, see Types.
kind	String	-	REST resource type corresponding to an object. The value, formatted in camel case, is derived from the endpoint and cannot be updated. For more information, see Resources.
metadata	Object	-	Kubernetes metadata, including namespaces and labels. For more information, see Metadata.
spec	Object	-	Specifications of the desired AscendJob status. replicaSpecs is a mandatory field.
spec.replicaSpecs	Object	-	Mapping from ReplicaType to ReplicaSpec, which specifies MS cluster configurations, for example, { "Scheduler": ReplicaSpec, "Worker": ReplicaSpec }.
spec.replicaSpecs.[ReplicaType]	Object	-	Replica description
spec.replicaSpecs.[ReplicaType].replicas	Integer	int32	Number of replicas required by the given template. The default value is 1.
spec.replicaSpecs.[ReplicaType].restartPolicy	String	-	Restart policy, including Always, OnFailure, Never, and ExitCode. The default value is Never.
spec.replicaSpecs.[ReplicaType].template	Object	-	Kubernetes pod template. For more information, see Kubernetes Pod Template.
spec.runPolicy	Object	-	Runtime policy (such as resource clearing and activity time) of a distributed training job.
spec.runPolicy.backoffLimit	Integer	int32	(Optional) Number of retries before a job fails.
spec.runPolicy.activeDeadlineSeconds	Integer	int64	Maximum duration (in seconds) for which a job keeps active. The value must be a positive integer. This field has no effect and will be deleted in later versions.
spec.runPolicy.cleanPodPolicy	String	-	Pod clearing policy after a job is complete. The default value is Running. This field has no effect and will be deleted in later versions.
spec.runPolicy.ttlSecondsAfterFinished	Integer	int32	Time to live (TTL) after a job is complete. By default, the value is infinite, but actual deletion may be delayed. This field has no effect and will be deleted in later versions.
spec.runPolicy.schedulingPolicy	Object	-	Scheduling policy, for example, gang-scheduling.
spec.runPolicy.schedulingPolicy.minAvailable	Integer	int32	Minimum number of available resources.
spec.runPolicy.schedulingPolicy.minResources	Object	-	Minimum resource set (integer or string) allocated by resource name.
spec.runPolicy.schedulingPolicy.priorityClass	String	-	Priority class name.
spec.runPolicy.schedulingPolicy.queue	String	-	Scheduling queue name.
spec.schedulerName	String	-	Scheduler specified when gang-scheduling is enabled. Currently, only Volcano is supported.
spec.successPolicy	String	-	Standard for marking AscendJob success. Currently, this field has no effect. A job is considered successful only when all pods are successful. This field will be deleted in later versions.
status	Object	-	Latest observed status of AscendJob (read-only). conditions and replicaStatuses are mandatory fields.
status.completionTime	String	date-time	Job completion time (RFC3339 format, UTC).
status.conditions	Array	-	Condition array for a job.
status.conditions[type]	String	-	Job condition type, for example, Complete.
status.conditions[status]	String	-	Condition status, including True, False, or Unknown.
status.conditions[lastTransitionTime]	String	date-time	Time when the condition status changes.
status.conditions[lastUpdateTime]	String	date-time	Last time after a condition is updated.
status.conditions[message]	String	-	Condition description.
status.conditions[reason]	String	-	Reason why a condition changes.
status.lastReconcileTime	String	date-time	Time when a job was last reconciled (RFC3339 format, UTC).
status.replicaStatuses	Object	-	Mapping from the replica type to the replica status.
status.replicaStatuses.[ReplicaType].active	Integer	int32	Number of running pods.
status.replicaStatuses.[ReplicaType].failed	Integer	int32	Number of failed pods.
status.replicaStatuses.[ReplicaType].succeeded	Integer	int32	Number of successful pods.
status.replicaStatuses.[ReplicaType].labelSelector	Object	-	Pod label selector (defining how to filter pods).
status.replicaStatuses.[ReplicaType].labelSelector.matchExpressions	Array	-	Label matching rule, supporting operators such as In, NotIn, Exists, and DoesNotExist.
status.replicaStatuses.[ReplicaType].labelSelector.matchLabels	Object	-	Key-value pairs of label matching (equivalent to the matchExpressions condition).
status.startTime	String	date-time	Job start time (RFC3339 format, UTC).

Job Status

After an acjob is started, you can run the kubectl get acjob command to view the running status of the acjob. The following table lists the status.

**Table 2** acjob running status
Status	Description
Created	A job has been created, but one or more sub-resources (pod/service) are not ready.
Running	All sub-resources (pod and service) of a job have been scheduled and started.
Restarting	One or more sub-resources (pod/service) of a job fail to run, but are being restarted according to the restart policy.
Succeeded	All sub-resources (pod and service) of a job are successfully terminated.
Failed	One or more sub-resources (pod or service) of a job fail to run.

Parent topic: API Reference