Link Establishment

Function Description

You can call LinkLlmClusters to establish communication links between Prefill and Decode nodes for KV cache transmission.

Scenario

Link establishment is a prerequisite for data transmission between nodes. The link establishment API adopts the TCP-like link establishment process. The server provides listening information during initialization, and the client initiates link establishment. The server/client roles are independent of the prompt/decoder roles and can be assigned as needed.

If the proportion of Prefill and Decode nodes in a cluster needs to be adjusted based on service load, you can add nodes by establishing links.

Example

Initializes the LLM-DataDist instance. The listening host IP address and port number must be set on the server.

// Server
LlmDataDist llm_data_dist(PROMPT_CLUSTER_ID, LlmRole::kPrompt);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
// Replace it with the actual IP address and port number.
options[OPTION_LISTEN_IP_INFO] = "ip:port";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}
// Client
LlmDataDist llm_data_dist(DECODER_CLUSTER_ID, LlmRole::kDecoder);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}

Call LinkLlmClusters on the client to initiate a link establishment operation.

std::vector<Status> rets;
std::vector<ClusterInfo> clusters;
ClusterInfo cluster_info;
IpInfo local_ip_info{};
// Replace it with the actual local IP address.
local_ip_info.ip = "ip";
IpInfo remote_ip_info{};
// Replace it with the actual peer IP address.
remote_ip_info.ip = "ip";
// Replace it with the port for listening used during initialization on the server.
remote_ip_info.port = "port";
cluster_info.remote_cluster_id = PROMPT_CLUSTER_ID;
cluster_info.local_ip_infos.emplace_back(std::move(local_ip_info));
cluster_info.remote_ip_infos.emplace_back(std::move(remote_ip_info));
clusters.emplace_back(std::move(cluster_info));
auto ret = llm_data_dist.LinkLlmClusters(clusters, rets);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] LinkLlmClusters failed, ret = %u\n", ret);
    return -1;
}
for (const auto &inner_ret : rets) {
    if (inner_ret != LLM_SUCCESS) {
        printf("[ERROR] LinkLlmClusters failed, ret = %u\n", inner_ret);
        return -1;
    }
}

Exception Handling

If LinkLlmClusters fails to be called, check whether the network connection between the two devices is normal, whether the host IP address is correct, and whether the port is occupied.

Parent topic: Link Management