Role Switchover

Scenario

In scenarios where the number of PD cluster nodes is fixed, roles can be switched between Prefill and Decode nodes as service demands change, optimizing resource utilization.

Related APIs

Call the SetRole API to switch the role of LLM-DataDist, and use options to switch between client and server roles.

Example

This example demonstrates switching from the Decoder role to the Prompt role, that is, from the client to the server. After switching the role, call other APIs of LLM-DataDist according to service requirements.

Initialize the LLM-DataDist instance.

LlmDataDist llm_data_dist(DECODER_CLUSTER_ID, LlmRole::kDecoder);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}

Call SetRole to switch roles. When switching from the client to the server, specify the listening host IP address and port number in options. When switching from the server to the client, do not specify the host IP address or port number.

std::map<AscendString, AscendString> options;
// Replace it with the actual IP address and port number.
options[OPTION_LISTEN_IP_INFO] = "ip:port";
llmDataDist.SetRole(LlmRole::kPrompt, options);

Exception Handling

For details about troubleshooting, see LLM-DataDist Error Codes.

Parent topic: Special Topics