InitModelForMultiPd

Function

Initializes models in the multi-server PD disaggregation scenario.

Format

Status InitModelForMultiPd(std::map<std::string, std::string> pdInfo, uint32_t modelInstanceId)

Parameters

Parameter	Mandatory/Optional	Description	Value
pdInfo	Mandatory	PD disaggregation information.	The value is of the std::map<std::string, std::string> type.
modelInstanceId	Mandatory	Model instance ID.	The value is of the uint32_t type. The value must be greater than 0 and defaults to 0.

Usage Example

Construct an LlmManager instance and initialize it.

// Define the function in 6.1.1.4.
mindie_llm::GetRequestsCallback getRequestCallback;
mindie_llm::SendResponsesCallback sendResponsesCallback;
mindie_llm::ControlSignalCallback stopSignalCallback;
mindie_llm::LlmManagerStatsCallback statsCallback;
mindie_llm::SendStatusResponseCallback sendStatusCallback;
const std::string llmConfigPath;
std::map<std::string, std::string> ipInfo;
std::shared_ptr<mindie_llm::LlmManager> llmManager = std::make_shared<mindie_llm::LlmManager>(llmConfigPath, getRequestCallback, sendResponsesCallback, stopSignalCallback, statsCallback, sendStatusCallback,ipInfo);
uint32 modelInstanceId = 0;
std::set<size_t> npuDeviceIds = {0};
llmManager->Init(modelInstanceId,npuDeviceIds); 
std::map<std::string, std::string> pdInfo;
LlmManager->InitModelForMultiPd(pdInfo, modelInstanceId)

Parent topic: LlmManager