Switch Affinity Scheduling 1.0

Instructions

• A switch cannot be used by multiple cross-switch jobs at the same time.
• The switches mentioned in this section are leaf switches by default.

Switch Affinity for Common Jobs

• If the number of replicas of a common job is less than M, M is the number of nodes connected to a leaf switch. Select preferentially nodes in a switch that meets the requirement for the number of job replicas and has fewer available nodes, then nodes that are not used by other switches, and finally nodes that do not cause downlink traffic congestion on the spine switch. Finally, perform random scheduling.
• If the number of replicas of a common job is greater than or equal to M, M is the number of nodes connected to a leaf switch. Select preferentially nodes in a switch that meets the requirement for the number of job replicas and has fewer available nodes and then nodes that do not cause downlink traffic congestion on the spine switch. Finally, perform random scheduling.

Switch Affinity for Foundation Model Jobs

• If the number of copies of a foundation model job is less than 4, the system selects preferentially nodes in a switch that meets the requirement for the number of job copies and has fewer available nodes and then nodes that are not used by other switches.
• If the number of replicas of a foundation model is greater than or equal to 4 and less than M, M is the number of nodes connected to a leaf switch. Select preferentially nodes in a switch that meets the requirement for the number of job replicas and has fewer available nodes and then nodes that do not cause downlink traffic congestion on the spine switch.
• If the number of replicas of a foundation model job is greater than or equal to M, M is the number of nodes connected to a leaf switch. Select preferentially nodes in a switch that meets the requirement for the number of job replicas and has fewer available nodes and then nodes that do not cause downlink traffic congestion on the spine switch.

Switch Affinity for Padding Jobs

Select preferentially nodes in a switch that meets the requirement for the number of job replicas and has fewer available nodes and then nodes that are not used by other switches.

Rescheduling Upon Faults

When the node where a job is located or Ascend AI processor is faulty, the job is rescheduled. Before rescheduling, pods of normal nodes will be scheduled to original nodes for training. Pods of faulty nodes will be rescheduled to other nodes.