Link Failover and Switchback

SwitchNicTrack

Description

Receives link failover requests from the O&M platform, and issues link failover/switchback operations to the device of the specified node of the training job. This interface needs to wait until the training job has run successfully and iterations have been produced before being called, to ensure that the job has been registered with ClusterD. The link failover/switchback interface is a manual O&M operation. For repeated switching scenarios, if each switching fails, it will cause frequent CKPT saving, posing a risk of disk space exhaustion.

Please issue the switchback command or link failover command only after the training iterations have become normal.

Function Prototype

rpc SwitchNicTrack(SwitchNics) returns (Status) {}

Input Parameters

Parameter Type (Protobuf Definition) Description

SwitchNics

Parameter	Type (Protobuf Definition)	Description
SwitchNics	message SwitchNics{ string jobID; map<string, DeviceList> nicOps; } message DeviceList { repeated string dev; repeated bool op; }	SwitchNics.jobID: Job ID. SwitchNics.nicOps: Devices and operations for which the user issues link failover/switchback Commands. The key is the node name, and the value is the Device to be operated on that node. DeviceList.dev: List of device IDs on this node. The number must match that of DeviceList.op. DeviceList.op: List of link failover operations to be performed for the devices corresponding to the DeviceIDs on this node. `true` indicates switching to the standby link, and `false` indicates using the primary link.

message SwitchNics{

string jobID;

map<string, DeviceList> nicOps;

}

message DeviceList {

repeated string dev;

repeated bool op;

}

SwitchNics.jobID: Job ID.

SwitchNics.nicOps: Devices and operations for which the user issues link failover/switchback Commands. The key is the node name, and the value is the Device to be operated on that node.

DeviceList.dev: List of device IDs on this node. The number must match that of DeviceList.op.

DeviceList.op: List of link failover operations to be performed for the devices corresponding to the DeviceIDs on this node. true indicates switching to the standby link, and false indicates using the primary link.

Return Value Description

Parameter Type (Protobuf Definition) Description

Status

Parameter	Type (Protobuf Definition)	Description
Status	message Status{ int32 code = 1; string info = 2; }	Status.code: Return code. Value `0`: Indicates that the command was issued successfully. Other values: Indicates that the issuance failed. Status.info: Return information.

message Status{

int32 code = 1;

string info = 2;

}

Status.code: Return code.

Value 0: Indicates that the command was issued successfully.
Other values: Indicates that the issuance failed.

Status.info: Return information.

SubscribeSwitchNicSignal

Description

Interface for the O&M platform to query the results of link failover/switchback. After the O&M personnel successfully issue an active link failover/switchback command, the results can be queried through this interface.

Prototype

rpc SubscribeSwitchNicSignal(SwitchNicRequest) returns (stream SwitchNicResponse) {}

Input Parameters

Parameter	Type (Protobuf Definition)	Description
SwitchNicRequest	message SwitchNicRequest{ string jobID; }	SwitchNicRequest.jobID: Job ID

Parameter

Type (Protobuf Definition)

Description

SwitchNicRequest

message SwitchNicRequest{

string jobID;

}

SwitchNicRequest.jobID: Job ID

Return Value Description

Parameter	Type (Protobuf Definition)	Description
SwitchNicResponse	message SwitchNicResponse{ string jobID; string msg; }	SwitchNicResponse.jobID: Job ID SwitchNicResponse.msg: Execution result of the link failover/switchback Command

Parameter

Type (Protobuf Definition)

Description

SwitchNicResponse

message SwitchNicResponse{

string jobID;

string msg;

}

SwitchNicResponse.jobID: Job ID

SwitchNicResponse.msg: Execution result of the link failover/switchback Command

SubscribeNotifySwitch

Description

Receives client subscription requests for link failover/switchback signals. The server assigns a message queue to each job and listens for pending messages in the queue. If messages exist, they are sent to the client via gRPC stream.

Prototype

rpc SubscribeNotifySwitch(ClientInfo) returns (stream SwitchRankList) {}

Input Parameters Description

Parameter	Type (Protobuf Definition)	Description
ClientInfo	message ClientInfo{ string jobId = 1; string role = 2; }	ClientInfo.jobId: Job ID. ClientInfo.role: Client role.

Parameter

Type (Protobuf Definition)

Description

ClientInfo

message ClientInfo{

string jobId = 1;

string role = 2;

}

ClientInfo.jobId: Job ID.

ClientInfo.role: Client role.

Sending Data

Parameter Type (Protobuf Definition) Description

SwitchRankList

Parameter	Type (Protobuf Definition)	Description
SwitchRankList	message SwitchRankList{ repeated string rankID = 1; repeated bool op = 2; string jobId = 3; }	SwitchRankList.rankID: List of device IDs on this node, with the same number of entries as `DeviceList.op`. SwitchRankList.op: The list of link failover operations to be performed for the devices corresponding to the device IDs on this node. `true` indicates switching to the standby link, `false` indicates using the primary link. SwitchRankList.jobId: Job ID

message SwitchRankList{

repeated string rankID = 1;

repeated bool op = 2;

string jobId = 3;

}

SwitchRankList.rankID: List of device IDs on this node, with the same number of entries as DeviceList.op.

SwitchRankList.op: The list of link failover operations to be performed for the devices corresponding to the device IDs on this node. true indicates switching to the standby link, false indicates using the primary link.

SwitchRankList.jobId: Job ID

Return Value

Parameter	Type (Protobuf Definition)	Description
stream	grpc stream	This interface returns a gRPC stream (the specific data structure of the return value depends on the programming language chosen by the client). The client can call the stream's `Receive` method (the specific method name depends on the programming language chosen by the client) to receive data pushed by the server.

ReplySwitchNicResult

Description

Interface for the client to return the link failover/switchback result to ClusterD.

Prototype

rpc ReplySwitchNicResult(SwitchResult) returns (Status) {}

Input Parameters

Parameter Type (Protobuf Definition) Description

SwitchResult

Parameter	Type (Protobuf Definition)	Description
SwitchResult	message SwitchResult{ string jobId = 1; bool result = 2; }	SwitchResult.jobId: Job ID. SwitchResult.result: Result of command execution: `true` for success, `false` for failure.

message SwitchResult{

string jobId = 1;

bool result = 2;

}

SwitchResult.jobId: Job ID.

SwitchResult.result: Result of command execution: true for success, false for failure.

Return Value

Parameter Type (Protobuf Definition) Description

Status

Parameter	Type (Protobuf Definition)	Description
Status	message Status{ int32 code = 1; string info =2; }	Status.code: Return code. Value `0`: Normal process Other values: Abnormal process Status.info: Return information.

message Status{

int32 code = 1;

string info =2;

}

Status.code: Return code.

Value 0: Normal process
Other values: Abnormal process

Status.info: Return information.