HandleLora

Function

Dynamically loads, unloads, and queries LoRA.

Format

Status HandleLora(const mindie_llm::LoraOperation loraOperation, std::vector<LoraParamSPtr> &loraInfo)

Parameters

Parameter	Mandatory/Optional	Description	Value
loraOperation	Mandatory	Selects a request for loading, unloading, or query.	mindie_llm::LoraOperation.
loraInfo	Mandatory	Information about the loaded, unloaded, or queried LoRA.	std::vector<LoraParamSPtr>.

Usage Example

Construct an LlmManager instance and deliver a LoRA loading request.

mindie_llm::GetRequestsCallback getRequestCallback;
mindie_llm::SendResponsesCallback sendResponsesCallback;
mindie_llm::ControlSignalCallback stopSignalCallback;
mindie_llm::LlmManagerStatsCallback statsCallback;
mindie_llm::SendStatusResponseCallback sendStatusCallback;
const std::string llmConfigPath;
std::map<std::string, std::string> ipInfo;
std::shared_ptr<mindie_llm::LlmManager> llmManager = std::make_shared<mindie_llm::LlmManager>(llmConfigPath, getRequestCallback, sendResponsesCallback, stopSignalCallback, statsCallback, sendStatusCallback,ipInfo);
std::shared_ptr<mindie_llm::InferRequest> request;
std::vector<LoraParamSPtr> loraInfo;
mindie_llm::LoraOPeration loraOperation = mindie_llm::LoraOPeration::LOAD;
llmManager->UpdateEngineInfo(loraOperation, loraInfo);

Return Value

Status object.

Parent topic: LlmManager