link_clusters

Applicability

Note: For the Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server, Atlas 300I A2 inference card, and A200I A2 Box heterogeneous components are supported.

Function Description

Establishes a link unilaterally. The request is initiated by the client. The server/client roles are independent of the prompt/decoder roles. Setting listen_ip_info on a node designates the node as the server (port listens for links).

Prototype

1

link_clusters(clusters: Union[List[LLMClusterInfo], Tuple[LLMClusterInfo]], timeout=3000)

Parameters

Example

Click Gitcode, select the matching version, and obtain the sample from the examples/python directory.

Returns

In normal cases, a tuple of two values is returned. The first value is the return value of the API, and the type is LLMStatusCode. The second value is the list of link setup results of each cluster, and the type is LLMStatusCode.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Restrictions

• The requirements for link establishment are as follows: Excessive links may cause OOM errors and deteriorate the transmission performance of the KV cache.
  • The maximum number of communication links that can be created is 512.
• You are advised to set the timeout interval longer than 200 ms.
• Register all memories before calling this API. Otherwise, memories registered after link establishment will not support remote access.
• In container scenarios, if local_comm_res is not configured or is set to an empty string, you must map the /etc/hccn.conf file into the container or ensure that hccn_tool exists in the default path /usr/local/Ascend/driver/tools. If neither condition is met, you need to add the directory containing hccn_tool to the PATH environment variable. A configuration example is provided below, where hccn_tool_install_path indicates the directory containing hccn_tool.

  export PATH=$PATH:${hccn_tool_install_path}

• If links are established concurrently and a device involved in the link establishment is in a down state, some link establishment processes may time out. To ensure successful link establishment using non-down devices, run the hccn_tool -i ${device_id} -tls -s enable 1 command for all devices involved in the link establishment.