Resumable Training APIs

taskd.python.toolkit.recover_module.recover_manager. DLRecoverManager (Internal, Do Not Call)

Function Description

The DLRecoverManager class provides APIs related to process-level recovery and process-level online recovery. The client imports it into the client code as a Python package.

init(self, info: pb.ClientInfo, server_addr: str)

Constructs a DLRecoverManager for subsequent communication.

Table 1 Parameters

register(self, request: pb.ClientInfo) -> pb.Status

Registers a client. The server performs pre-recovery initialization operations for the job specified by the request.

Table 2 Parameters

Table 3 Return value description

def start_subscribe(self, frame: str = "pytorch")

The client and server establish a gRPC persistent connection, through which the server communicates unidirectionally with the client. For example, when a fault occurs, the server sends the client information such as a training stop signal and global faulty process rank.

Table 4 Parameters

init_clusterd(self)

Initializes the ClusterD server status on the client side to ensure normal registration and connection establishment for subsequent jobs.

report_stop_complete(code: int, msg: str, fault_ranks: dict) -> int

Function Description

The client reports to the server that the job process has stopped. Typically, after the client receives a stop training signal from the server, the client stops the training job process and then reports to the server that the job process has stopped.

Input Parameters

Table 1 Parameter description

Return Value

Table 2 Return value description

report_recover_strategy(fault_ranks: dict, strategy_list: list) -> int

Function Description

The client provides the server with the recovery strategies it supports, so that the server can select the optimal recovery strategy. The server then sends it to the client through the persistent connection established by start_subscribe.

Input Parameters

Table 1 Parameter Description

Return Value

Table 2 Return value description

report_recover_status(code: int, msg: str, fault_ranks: dict, strategy: str) -> int

Function Description

The client reports the job recovery status to the server.

Input Parameters

Table 1 Parameter description

Return Value

Table 2 Return value description

report_process_fault(fault_ranks: dict) -> int

Function Description

The client reports a service-plane fault in the job process. When the client detects a fault first, it reports the information of the rank where the service-plane fault occurred to the server.

Input Parameters

Table 1 Parameter description

Return Value

Table 2 Return value description

taskd.python.framework.agent.ms_mgr.msrun_plugin. MSRunPlugin

The MSRunPlugin class provides MindSpore process management functions. It is called by MindSpore and integrated into the MindSpore package.

register_callbacks(self, operator, func)

Function Description

Registers a process management function with TaskD for subsequent use in managing the process lifecycle.

Input Parameters

Table 1 Parameter description

start(self)

Calls the MSRunPlugin start method to allow TaskD to take over MindSpore training process management.

init(self)

Constructs the MSRunPlugin class for subsequent instantiation and invocation by the user.