LlmManager

Function

Default constructor

C++ Prototype

LlmManager(const std::string &llmConfigPath, MindIE_LLM::GetRequestsCallback getRequest,
        MindIE_LLM::SendResponsesCallback sendResponse, MindIE_LLM::ControlSignalCallback controlCallback,
        MindIE_LLM::LlmManagerStatsCallback statusCallback,
        MindIE_LLM::SendStatusResponseCallback statusResponseCallback)

Python Prototype

LlmManager(config_path, get_request, send_response, control_callback, status_callback, status_response_callback)

Parameters

Parameter	Mandatory/Optional	Description	Value
config_path	Mandatory	Config path	Character string, which must be a valid config path.
get_request	Mandatory	GetRequest callback function	The output of Callable[[], List['InferRequest']] is a list of the InferRequest type.
send_response	Mandatory	SendResponses callback function	Callable[[InferRequestId, TensorMap, bool, str], None] The inputs are of the InferRequestId, TensorMap, bool, and string types in sequence. There is no output.
control_callback	Mandatory	ControlSignal callback function	Callable[[InferRequestId, Status, int], None] The inputs are of the InferRequestId type, character type, Status type, and StatusResponseType type in sequence. There is no output.
status_callback	Mandatory	LlmManagerStats callback function	Callable[[], Set[InferRequestId]] There is no input. The output is a set of InferRequestId.
status_response_callback	Mandatory	SendStatusResponse callback function	Callable[[str], None] The input is of the string type, and there is no output.

Usage Example

LlmManager initialization constructor, which passes the configuration file path and callback functions.

config_path = '/path/to/config.json'
def get_requests_callback():
    ...
    pass
def send_response_callback():
    ...
    pass
def control_callback():
    ...
    pass
def status_callback():
    ...
    pass
def status_response_callback():
    ...
    pass

llm_manager = LlmManager(config_path, get_requests_callback, send_response_callback, control_callback, status_callback, status_response_callback)

Parent topic: LlmManager