Role Switchover

Application Scenarios

In scenarios where the number of PD cluster nodes is fixed, roles can be switched between Prefill and Decode nodes as service demands change, optimizing resource utilization.

Related APIs

Call the SetRole API to switch the role of LLM-DataDist.

Example

This example demonstrates switching the role from Decoder to Prompt. After switching the role, call other APIs of LLM-DataDist according to service requirements.

During LLM-DataDist initialization, set OPTION_ENABLE_SET_ROLE to "1".

LlmDataDist llm_data_dist(DECODER_CLUSTER_ID, LlmRole::kDecoder);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
options[OPTION_BUF_POOL_CFG] = R"({
"buf_pool_size": 2147483648
})";
options[OPTION_ENABLE_SET_ROLE] = "1";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}

Call SetRole to switch the role.

std::map<AscendString, AscendString> options;
// Replace it with the actual IP address and port number.
options[OPTION_LISTEN_IP_INFO] = "ip:port";
llmDataDist.SetRole(LlmRole::kPrompt, options);

Troubleshooting

For details about troubleshooting, see Error Codes.

Parent topic: Special Topics