switch_role

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Switches the role of the current LLM-DataDist instance. You are advised to use this function only when PagedAttention is enabled.

Prototype

switch_role(self, role: LLMRole, switch_options: Optional[Dict[str, str]] = None)

Parameters

Parameter	Data Type	Description
role	LLMRole	Target role to be switched.
switch_options	options: Dict[str, str]	Role switching configuration options. This parameter is optional. The default value is None. This parameter must be configured when the role is switched to Prompt, and its value shall include listen_ip_info.

Parameter

Data Type

Description

role

LLMRole

Target role to be switched.

switch_options

options: Dict[str, str]

Role switching configuration options.

This parameter is optional. The default value is None.

This parameter must be configured when the role is switched to Prompt, and its value shall include listen_ip_info.

Example

from llm_datadist import LLMDataDist, LLMRole
llm_datadist = LLMDataDist(LLMRole.DECODER, 0)
...
switch_options = { 'llm.listenIpInfo': '127.0.0.1:26000' }
llm_datadist.switch_role(LLMRole.PROMPT, switch_options)

Returns

In normal cases, no value is returned.
If the input data type is incorrect, a TypeError or ValueError is thrown.
If enable_switch_role is not enabled in LLMConfig during LLM-DataDist initialization, an LLMException is thrown when this API is called, and status_code is LLM_FEATURE_NOT_ENABLED.
If there are residual link resources during switch_role, an LLMException is thrown, and status_code is LLM_EXIST_LINK.
If the target role of switch_role is the same as the current role, an LLMException is thrown, and status_code is LLM_PARAM_INVALID.
This API cannot be called in single-process multi-device mode.

Constraints

None

Parent topic: LLMDataDist