Performance Degradation

ModifyTrainingDataTraceSwitch

Description

External call to modify the capability of various dynamic data tracing.

If lightweight profiling is enabled or modified through the gRPC interface provided by ClusterD to obtain data written to disk, the lifecycle of the created data-trace-<Job Name> ConfigMap will be deleted along with the job. When the job does not exist, this interface call will fail.

Prototype

rpc ModifyTrainingDataTraceSwitch(DataTypeReq) returns (DataTypeRes) {}

Input Parameters

ParameterType (Protobuf Definition)Description
DataTypeReq

message DataTypeReq{

string jobNsName = 1;

ProfilingSwitch profilingSwitch = 2;

}

message ProfilingSwitch{

string CommunicationOperator = 1;

string Step = 2;

string SaveCheckpoint = 3;

string FP =4;

string DataLoader =5;

}

jobNsName: The namespace and name of the job to be modified, concatenated with '/', for example: default/test-pytorch.

profilingSwitch: Details of various switches.

  • CommunicationOperator: Communication operators.
  • Step: Step latency.
  • SaveCheckpoint: SaveCheckpoint time consumption.
  • FP: Forward propagation.
  • DataLoader: DataLoader time consumption.

Return Value

ParameterType (Protobuf Definition)Description
DataTypeResmessage DataTypeRes{

string message = 1;

int32 code = 2;

}

message: Interface call result.

code: Interface call return code.

  • 300: invalid input parameters.
  • 404: unable to query ConfigMap.
  • 500: server exception.
  • 200: interface returned normally.

GetTrainingDataTraceSwitch

Description

External call to obtain the status of various dynamic data tracing.

Prototype

rpc GetTrainingDataTraceSwitch(DataStatusReq) returns (DataStatusRes) {}

Input Parameters

ParameterType (Protobuf Definition)Description
DataStatusReqmessage DataStatusReq{

string jobNsName = 1;

}
jobNsName: The namespace and name of the job to be modified, concatenated with '/', e.g., default/test-pytorch.

Return Value Description

ParameterType (Protobuf Definition)Description
DataStatusResmessage DataStatusRes{

string message = 1;

ProfilingSwitch profilingSwitch = 2;

int32 code = 3;

}

message: Result information of the interface call.

profilingSwitch: Details of various switches.

  • CommunicationOperator: Communication operator.
  • Step: Step latency.
  • SaveCheckpoint: SaveCheckpoint time consumption.
  • FP: Forward propagation.
  • DataLoader: DataLoader time consumption.
code: Return code of the interface call.
  • 300: invalid input parameters.
  • 404: unable to query ConfigMap.
  • 500: server exception.
  • 200: interface returned normally.

SubscribeDataTraceSwitch

Description

Externally subscribe to the status of various dynamic data tracing switches.

Prototype

rpc SubscribeDataTraceSwitch(ProfilingClientInfo) returns (stream DataStatusRes) {}

Input Parameters

ParameterType (Protobuf Definition)Description
ProfilingClientInfomessage ProfilingClientInfo{

string jobId = 1;

string role = 2;

}

jobId: Job ID.

role: Client role.

Return Value Description

ParameterType (Protobuf Definition)Description
DataStatusResmessage DataStatusRes{

string message = 1;

ProfilingSwitch profilingSwitch = 2;

int32 code = 3;

}

message: Information about the interface call result.

profilingSwitch: Details of various switches.

  • CommunicationOperator: Communication operator.
  • Step: Step latency.
  • SaveCheckpoint: SaveCheckpoint time consumption.
  • FP: Forward propagation.
  • DataLoader: DataLoader time consumption.
code: Return code of the interface call.
  • 300: invalid input parameters.
  • 404: unable to query ConfigMap.
  • 500: server exception.
  • 200: interface returned normally.