[object Object]
Obtains the Elastic Agent version number.
Input: Empty
Return Value: Elastic Agent version number
Usage example:
Function[object Object][object Object]
When building a training image and installing Elastic Agent, after using the command [object Object], Elastic Agent executes the [object Object] interface when importing Torch's Elastic module, so that the patch takes effect automatically.
Elastic Agent patches methods such as [object Object], [object Object], and [object Object] to additionally provide fault detection and recovery capabilities for Ascend NPU devices.
The [object Object] class provides interfaces for process-level recovery and process-level online recovery. The client imports it into the client code as a Python package.
[object Object]
init(self, info: pb.ClientInfo, server_addr: str, secure_conn: bool = True, cert_path: str = "")[object Object][object Object]
Constructs a [object Object] for subsequent communication.
Table 1 Parameters
register(self, request: pb.ClientInfo) -> pb.Status[object Object][object Object]
Registers the client. The server performs pre-recovery initialization for the job specified by [object Object].
Table 2 Parameters
Table 3 Return values
start_subscribe(self)[object Object][object Object]
The client and server establish a gRPC persistent connection, through which the server communicates unidirectionally with the client. For example, when a fault occurs, the server sends the client information such as stop training signal and global fault process rank.
init_clusterd(self)[object Object][object Object]
The client initializes the ClusterD server status to ensure that subsequent jobs can be registered and connections established normally.
The client reports to the server that the job process has stopped. Typically, after receiving a training stop signal from the server, the client stops the training job process and then reports the process stop completion to the server.
Table 1 Parameters
Table 2 Return values
The client reports the recovery policies it supports to the server, so that the server can select the optimal recovery policy. The server then sends it to the client through the persistent connection established by [object Object].
Table 1 Parameters
Table 2 Return values
The client reports the job recovery status to the server.
Table 1 Parameters
Table 2 Return values
The client reports a service-plane fault of the job process. When the client detects a fault first, it reports the information about the rank where the service-plane fault is located to the server.
Table 1 Parameters
Table 2 Return values
The Elastic-Agent return codes are shown in .
Table 1 Elastic-Agent return codes
[object Object][object Object]