Link Disconnection

Function Description

You can call UnLinkLlmClusters to disconnect and clean up the communication links between Prefill and Decode nodes.

Scenario

When a node in the Prefill or Decode cluster is abnormal, you can clean up the abnormal link by disconnecting it. If the ratio of Prefill and Decode nodes in the cluster needs to be adjusted, disconnect the established links.

Example

There are two disconnection methods:

Initiate a disconnection request on the client, which is commonly used in non-faulty link scenarios.

// clusters must be the same as those used for link establishment.
auto ret = llm_data_dist.UnLinkLlmClusters(clusters, rets);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] UnLinkLlmClusters failed, ret = %u\n", ret);
    return -1;
}
for (const auto &inner_ret : rets) {
    if (inner_ret != LLM_SUCCESS) {
        printf("[ERROR] UnLinkLlmClusters failed, ret = %u\n", inner_ret);
        return -1;
    }
}

Force link disconnection from both the Prefill and Decode sides, which is typically used in link fault scenarios.

// Prefill and Decode sides
std::vector<Status> rets;
std::vector<ClusterInfo> clusters;
ClusterInfo cluster_info;
IpInfo local_ip_info;
// Replace it with the actual local IP address.
local_ip_info.ip = "ip";
IpInfo remote_ip_info;
// Replace it with the actual peer IP address.
remote_ip_info.ip = "ip";
// Replace it with the port for listening used during initialization on the server.
remote_ip_info.port = "port";
cluster_info.remote_cluster_id = PROMPT_CLUSTER_ID;
cluster_info.local_ip_infos.emplace_back(std::move(local_ip_info)); // local_ip_infos stores the local host IP addresses.
cluster_info.remote_ip_infos.emplace_back(std::move(remote_ip_info)); // remote_ip_infos stores the peer host IP addresses.
clusters.emplace_back(std::move(cluster_info));
auto ret = llmDataDist.UnLinkLlmClusters(clusters, rets, 1000, true);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] UnLinkLlmClusters failed, ret = %u\n", ret);
    return -1;
}
for (const auto &inner_ret : rets) {
    if (inner_ret != LLM_SUCCESS) {
        printf("[ERROR] UnLinkLlmClusters failed, ret = %u\n", inner_ret);
        return -1;
    }
}

Exception Handling

If UnLinkLlmClusters fails to be called due to a network fault, forcibly disconnect the link.

Parent topic: Link Management