Class Introduction

Function

This class establishes a client for interacting with LLM services and facilitating communication with LLMs. Currently, it supports only the OpenAI API /v1/chat/completions. This class inherits and implements langchain.llms.base.LLM.

Prototype

from mx_rag.llm import Text2TextLLM
# All parameters must be passed through keyword parameters.
Text2TextLLM(base_url, model_name, llm_config, client_param)

Parameters

Parameter	Data Type	Required/Optional	Description
base_url	str	Required	Service address of an LLM. The length range is [1, 128].
model_name	str	Required	LLM name. The length range is [1, 128].
llm_config	LLMParameterConfig	Optional	This parameter is valid when for LangChain-based call. For details, see LLMParameterConfig. For non-LangChain-based call, parameters are passed via chat and chat_streamly. For details, see chat and chat_streamly.
client_param	ClientParam	Optional	HTTPS client configuration parameter. The default value is ClientParam(). For details, see ClientParam.

Example

from mx_rag.llm import Text2TextLLM, LLMParameterConfig
from mx_rag.utils import ClientParam
llm = Text2TextLLM(base_url="https://{ip}:{port}/v1/chat/completions",
                   model_name="qianwen-7b",
                   llm_config=LLMParameterConfig(max_tokens=512),
                   client_param=ClientParam(ca_file="/path/to/ca.crt")
                   )
res = llm.chat("Please introduce Beijing.")
print(res)
for res in llm.chat_streamly("Please introduce Beijing."):
    print(res)

Parent topic: Text2TextLLM