link_clusters

Applicable Products

Note: For the Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Initiates link establishment unilaterally from the client. Since dynamic scaling primarily occurs on the Decode side, the Prefill side is defined as the server and the Decode side as the client. Consequently, link establishment follows a procedure where the Decode side initiates link requests to the Prefill side.

Prototype

1

link_clusters(clusters: Union[List[LLMClusterInfo], Tuple[LLMClusterInfo]], timeout=3000)

Parameters

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from llm_datadist import LLMDataDist, LLMRole, LLMStatusCode, LLMClusterInfo
llm_datadist = LLMDataDist(LLMRole.DECODER, 0)
cluster = LLMClusterInfo()
cluster.remote_cluster_id = 1
cluster.append_local_ip_info("1.1.1.1", 26000)
cluster.append_remote_ip_info("1.1.1.1", 26000)
ret, rets = llm_datadist.link_clusters([cluster], 5000)
if ret != LLMStatusCode.LLM_SUCCESS:
    raise Exception("link failed.")
for cluster_i in range(len(rets)):
    link_ret = rets[cluster_i]
    if link_ret != LLMStatusCode.LLM_SUCCESS:
        print(f"{cluster_i} link failed.")

Returns

In normal cases, a tuple of two values is returned. The first value is the return value of the API, and the type is LLMStatusCode. The second value is the list of link setup results of each cluster, and the type is LLMStatusCode.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Constraints

• The requirements for link establishment are as follows: Excessive links may cause out-of-memory (OOM) errors and deteriorate the transmission performance of the KV cache.
  • The number of concurrent connections on the server side (Prefill side) is 16, and the maximum number of connections that can be created is 512.
  • The maximum number of communication connections that can be created on the client side (Decode side) is 512.
• It is recommended that the timeout period be set to a value greater than 200 ms.
• The call must be initiated by the client.