Restoring Parameter Passing on the Parameter Plane

(Optional) Disabling Parameter Passing Recovery on the Parameter Plane

For the process-level rescheduling and process-level online recovery features, if you want to disable this function and load parameters from the storage checkpoint, you need to modify the job YAML file. The following is an example of using process-level rescheduling and disabling parameter passing recovery on the parameter plane.

...  
metadata: 
   labels:  
     ...  
     fault-scheduling: "grace"
 ... 
...  
   annotations:  
     ...  
     recover-strategy: "recover" # Recovery policy. recover indicates that process-level rescheduling is enabled.
... 
...
spec:
  replicaSpecs:
    Master:
      template:
        spec:
          containers:
          - name: ascend # do not modify
            ...
            args:
              - | 
                ...
                bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                  ...
                  --distributed-optimizer-no-replica \
                  ...
    Worker:
      template:
        spec:
          containers:
          - name: ascend # do not modify
            ...
            args:
              - |
                ...
                bash scripts/train_start.sh /job/code /job/output pretrain_gpt.py \
                  ...
	          --distributed-optimizer-no-replica \
                  ...
...