Documentation
Download
文档中心
Process-Level Recovery APIs

Process-Level Recovery

gRPC Interface

Register (Public Interface)

Description

Receives and processes client registration requests, and prepares initialization for process-level recovery. After the job successfully calls Init, it must wait for the client to confirm that the process-level rescheduling and process-level online recovery on the MindIO side are enabled before calling this interface. After Register succeeds, the process-level rescheduling and process-level online recovery functions become available.

Prototype

rpc Register(ClientInfo) returns (Status) {}

Input Parameters

ParameterType (Protobuf Definition)Description
ClientInfomessage ClientInfo{

string jobId = 1;

string role = 2;

}

ClientInfo.jobId: Job ID

ClientInfo.role: Client role

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

  • 0: Registration successful.
  • Other values: Registration failed.

Status.info: Return information.

Init

Description

Used to initialize process-level rescheduling and process-level online recovery. After successful initialization, the process-level rescheduling and process-level online recovery functions will be temporarily unavailable.

Prototype

rpc Init(ClientInfo) returns (Status) {}

Input Parameters

ParameterType (Protobuf Definition)Description
ClientInfomessage ClientInfo{

string jobId = 1;

string role = 2;

}

ClientInfo.jobId: Job ID

ClientInfo.role: Client role

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

  • 0: Registration successful.
  • Other values: Registration failed.

Status.info: Return information.

SubscribeProcessManageSignal

Description

Receives client subscription requests for process control signals. The server allocates a message queue for each job and listens for pending messages in the queue. If a message exists, it is sent to the client via the gRPC stream.

Prototype

rpc SubscribeProcessManageSignal(ClientInfo) returns (stream ProcessManageSignal){}

Input Parameters

ParameterType (Protobuf Definition)Description
ClientInfomessage ClientInfo{

string jobId = 1;

string role = 2;

}

ClientInfo.jobId: Job ID.

ClientInfo.role: Client role.

Sending Data Description

ParameterType (Protobuf Definition)Description
ProcessManageSignal

message FaultRank{

string rankId = 1;

string faultType = 2;

}

message ProcessManageSignal{

string uuid=1;

string jobId = 2;

string signalType = 3;

repeated string actions = 4;

repeated FaultRank faultRanks = 5;

string changeStrategy = 6;

int64 timeout = 7;

}

rankId: String, ID of the faulty rank

faultType: String, fault type

uuid: String, UUID of this signal

jobId: String, training job ID

signalType: String, signal type

actions: Repeated string, actions to be executed

faultRanks: Repeated FaultRank, information about faulty ranks

changeStrategy: String, recovery strategy to be executed

timeout: int64, timeout duration

Return Value

Return ValueType (Protobuf Definition)Description
streamgrpc stream
  • This interface returns a gRPC stream (the specific data structure of the return value depends on the programming language chosen by the client).
  • The client can call the stream's Receive method (the specific method name depends on the programming language chosen by the client) to receive data pushed by the server.
nodeRankIdsstring arrayNode Rank ID of the faulty node.
extraParamsstringPasses specific scaling policy information in the form of a JSON string, which is transparently transmitted to MindIO via TaskD and ultimately passed to the callback function for parsing.

ReportStopComplete

Description

Receives the report from the client on whether the training process was successfully paused.

Prototype

rpc ReportStopComplete(StopCompleteRequest) returns (Status){}

Input Parameters

ParameterType (Protobuf Definition)Description
StopCompleteRequestmessage StopCompleteRequest{

string jobId = 1;

Status status = 2;

repeated FaultRank faultRankIds = 3;

}

StopCompleteRequest.jobId: Job ID.

StopCompleteRequest.status.code: Return code. OK indicates that the training process was paused successfully; other values indicate that the pause failed.

StopCompleteRequest.status.info: Return information.

StopCompleteRequest.faultRankIds: List of global fault ranks for the faulty chips. FaultRank is a set of key-value pairs containing fault information, consisting of rankId (global rank ID) and faultType (fault type). faultType = 0 indicates an on-chip Memory fault; faultType = 1, it indicates other faults; faultType = 2 indicates a network fault.

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

  • 0: the fault recovery process is normal.
  • Other values: the fault recovery process is abnormal, triggering rescheduling.

Status.info: Return information.

ReportRecoverStrategy

Description

Receives the fault recovery strategy supported by the current job reported by the client.

Prototype

rpc ReportRecoverStrategy(RecoverStrategyRequest) returns (Status) {}

Input Parameters

ParameterType (Protobuf Definition)Description
RecoverStrategyRequestmessage RecoverStrategyRequest{

string jobId = 1;

repeated FaultRank faultRankIds = 2;

repeated string strategies = 3;

}

RecoverStrategyRequest.jobId: Job ID

RecoverStrategyRequest.faultRankIds: List of global fault ranks for faulty chips. FaultRank is a set of key-value pairs containing fault information, consisting of rankId (global rank ID) and faultType (fault type). faultType = 0 indicates an on-chip Memory fault; faultType = 1, it indicates other faults; faultType = 2 indicates a network fault.

RecoverStrategyRequest.strategies: Recovery strategies supported by the Current job.

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return Code.

  • 0: the fault recovery process is normal.
  • Other values: the recovery process is abnormal, triggering rescheduling.

Status.info: Return information.

ReportRecoverStatus

Description

Receives the current job recovery status reported by the client.

Prototype

rpc ReportRecoverStatus(RecoverStatusRequest) returns (Status) {}

Input Parameters

ParameterType (Protobuf Definition)Description
RecoverStatusRequestmessage RecoverStatusRequest{

string jobId = 1;

Status status = 2;

string strategy = 3;

repeated string isolateRankIds = 4;

}

RecoverStatusRequest.jobId: Job ID.

RecoverStatusRequest.status.code: Job recovery status code.

  • 0: job recovery is successful.
  • Other values: job recovery failed.

RecoverStatusRequest.status.info: Job recovery status.

RecoverStatusRequest.strategy: Recovery strategy name.

RecoverStatusRequest.isolateRankIds: List of ranks to be isolated when MindIO reports scale-in messages.

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

  • 0: the fault recovery process is normal.
  • Other values: the recovery process is abnormal, triggering rescheduling.

Status.info: Return information.

ReportProcessFault

Description

Receives the global rank information of the faulty chips reported by the client.

Prototype

rpc ReportProcessFault(ProcessFaultRequest) returns (Status){}

Input Parameters

ParameterType (Protobuf Definition)Description
ProcessFaultRequestmessage ProcessFaultRequest{

string jobId = 1;

repeated FaultRank faultRankIds = 2;

}

ProcessFaultRequest.jobId: Job ID.

ProcessFaultRequest.faultRankIds: List of global rank IDs of faulty chips. FaultRank is a set of key-value pairs containing fault information, consisting of rankId (global rank ID) and faultType (fault type). faultType = 0 indicates an on-chip Memory fault; faultType = 1, it indicates other faults; faultType = 2 indicates a network fault.

Return Value

Return ValueType (Protobuf Definition)Description
Statusmessage Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

  • 0: the recovery process is normal.
  • Other values: the fault recovery process is abnormal, triggering rescheduling.

    • Status.info: Return information.

    HealthCheck

    Description

    Checks the gRPC connection status.

    Prototype

    rpc HealthCheck(ClientInfo) returns (Status) {}

    Input Parameters

    ParameterType (Protobuf Definition)Description
    ClientInfomessage ClientInfo{

    string jobId = 1;

    string role = 2;

    }

    ClientInfo.jobId: Job ID.

    ClientInfo.role: Client role.

    Return Value

    Return ValueType (Protobuf Definition)Description
    Statusmessage Status{

    int32 code = 1;

    string info = 2;

    }

    Status.code: Return code.

  • 0: the fault recovery process is normal.
  • Other values: the fault recovery process is abnormal, triggering rescheduling.

    • Status.info: Return information.

    Interfaces for Integrating with Third-Party AI Platforms

    Description

    An AI platform can control the fault recovery process and recovery strategy through Pod Group Annotations. For example, when the platform writes the Pod Group Annotation key: ProcessRecoverStrategy with an empty value, the fault recovery will be blocked until the platform writes a specific recovery strategy to continue the recovery process.

    Pod Group Annotation

    Table 1 Parameters

    ParameterValueDescription
    ProcessRecoverStrategy
    • retry
    • recover
    • dump
    • Empty or none
    • Field does not exist
    • retry: The platform initiates recovery, with the strategy being process-level online recovery
    • recover: The platform initiates recovery, with the strategy being online recovery
    • dump: The platform initiates recovery, with the strategy being saving dying gasps
    • Empty or none: Waiting for platform decision
    • Field does not exist: Disable process-level recovery
    ProcessConfirmFaultstringA list of fault key-value pairs refreshed by ClusterD, formatted as a string of "id1:type1,id2:type2". id represents the global rank ID, and type represents the fault type. type = 0 indicates that the faulty chip only has on-chip memory faults, and type = 1 indicates at least one non-on-chip memory fault.
    ProcessResultFaultstringA list of fault key-value pairs confirmed by the platform, formatted as a string of "id1:type1,id2:type2". id represents the global rank ID, and type represents the fault type. type = 0 indicates that the faulty chip only has on-chip memory faults, and type = 1 indicates at least one non-on-chip memory fault.
    RankTableReady
    • true
    • false or other values
    • Field does not exist
    • true: The platform has generated the RankTable
    • false or other values: The platform has not yet generated the RankTable
    • Field does not exist: Non-RankTable mode
    ProcessRecoverStatus
    • retry-success
    • retry-failed
    • recover-success
    • recover-failed
    • dump-success
    • dump-failed
    • exit-completed
    • Empty or other values
    • retry-success: Process-level online recovery succeeded
    • retry-failed: Process-level online recovery failed
    • recover-success: Online recovery succeeded
    • recover-failed: Online recovery failed
    • dump-success: Saving dying gasps succeeded
    • dump-failed: Saving dying gasps failed
    • exit-completed
    • Empty or other values: Recovery not completed
    This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details