link_clusters

Applicability

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Initiates link establishment unilaterally from the client. Since dynamic scaling primarily occurs on the Decode side, the Prefill side is defined as the server and the Decode side as the client. Consequently, link establishment follows a procedure where the Decode side initiates link requests to the Prefill side.

Prototype

1

link_clusters(clusters: Union[List[LLMClusterInfo], Tuple[LLMClusterInfo]], timeout=3000)

Parameters

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from llm_datadist import LLMDataDist, LLMRole, LLMStatusCode, LLMClusterInfo
llm_datadist = LLMDataDist(LLMRole.DECODER, 0)
cluster = LLMClusterInfo()
cluster.remote_cluster_id = 1
cluster.append_local_ip_info("1.1.1.1", 26000)
cluster.append_remote_ip_info("1.1.1.1", 26000)
ret, rets = llm_datadist.link_clusters([cluster], 5000)
if ret != LLMStatusCode.LLM_SUCCESS:
    raise Exception("link failed.")
for cluster_i in range(len(rets)):
    link_ret = rets[cluster_i]
    if link_ret != LLMStatusCode.LLM_SUCCESS:
        print(f"{cluster_i} link failed.")

Returns

In normal cases, a two-element tuple is returned. The first element is the return value of the API, of type LLMStatusCode. The second element is a list of link establishment results of each cluster, also of type LLMStatusCode.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Constraints

• The requirements for link establishment are as follows: Excessive links may cause out-of-memory (OOM) errors and deteriorate the transmission performance of the KV cache.
  • The number of concurrent connections on the server side (Prefill side) is 16, and the maximum number of connections that can be created is 512.
  • The maximum number of communication connections that can be created on the client side (Decode side) is 512.
• It is recommended that the timeout period be set to a value greater than 200 ms.
• The call must be initiated by the client.