Link Establishment

Function Description

You can call LinkLlmClusters to establish communication links between Prefill and Decode nodes for KV cache transmission.

Application Scenarios

Link establishment is a prerequisite for data transmission between nodes. In D2D mode, the link establishment API adopts the TCP-like link establishment process, where the Prefill side provides listening information during initialization and the Decode side initiates link establishment.

If the proportion of Prefill and Decode nodes in a cluster needs to be adjusted based on service load, you can add nodes by establishing links.

Example

Initialize the LLM-DataDist instance. The device IP address and port for listening need to be set on the Prefill side.

// Prefill
LlmDataDist llm_data_dist(PROMPT_CLUSTER_ID, LlmRole::kPrompt);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
// Replace it with the actual IP address and port number.
options[OPTION_LISTEN_IP_INFO] = "ip:port";
options[OPTION_BUF_POOL_CFG] = R"({
"buf_cfg":[{"total_size":2097152,"blk_size":256,"max_buf_size":8192}],
"buf_pool_size": 2147483648
})";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}
// Decode
LlmDataDist llm_data_dist(DECODER_CLUSTER_ID, LlmRole::kDecoder);
std::map<AscendString, AscendString> options;
options[OPTION_DEVICE_ID] = "0";
options[OPTION_BUF_POOL_CFG] = R"({
"buf_pool_size": 2147483648
})";
auto ret = llm_data_dist.Initialize(options);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] Initialize failed, ret = %u\n", ret);
    return -1;
}

Call LinkLlmClusters on the Decode side to initiate a link establishment operation.

std::vector<Status> rets;
std::vector<ClusterInfo> clusters;
ClusterInfo cluster_info;
IpInfo local_ip_info;
// Replace it with the actual local IP address.
local_ip_info.ip = "ip";
IpInfo remote_ip_info;
// Replace it with the actual peer IP address.
remote_ip_info.ip = "ip";
// Replace it with the port for listening used during initialization on the Prefill side.
remote_ip_info.port = "port";
cluster_info.remote_cluster_id = PROMPT_CLUSTER_ID;
cluster_info.local_ip_infos.emplace_back(std::move(local_ip_info));
cluster_info.remote_ip_infos.emplace_back(std::move(remote_ip_info));
clusters.emplace_back(std::move(cluster_info));
auto ret = llm_data_dist.LinkLlmClusters(clusters, rets);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] LinkLlmClusters failed, ret = %u\n", ret);
    return -1;
}
for (const auto &inner_ret : rets) {
    if (inner_ret != LLM_SUCCESS) {
        printf("[ERROR] LinkLlmClusters failed, ret = %u\n", inner_ret);
        return -1;
    }
}

Troubleshooting

If LinkLlmClusters fails to be called, check whether the network connection between the two devices is normal, whether the device IP address is correct, and whether the port is occupied.

For more troubleshooting methods, see Error Codes.

Parent topic: Link Management