[object Object]

Low-Rank Adaptation (LoRA) is an efficient parameter fine-tuning method. It decomposes the weight matrix of an LLM into the original weight matrix and a product of two low-rank matrices, that is, W' = W + BA. The number of parameters required for training matrix B and matrix A is far less than that required by the original weights. Therefore, the product result can be integrated into the linear layer and passed downwards, achieving lightweight fine-tuning of LLMs.

Multi-LoRA uses multiple LoRA weights for inference based on a base model. Each request carries a specified LoRA ID. During inference, the corresponding LoRA weight is dynamically matched. During service deployment, the LoRA weights and base model weights are pre-loaded to the graphics memory. A single inference request supports zero or at most one LoRA weight. For models with a large number of parameters, if the parameters cannot fit on a single device, tensor parallelism can be used.

The LoRA weight files must include the [object Object] and [object Object] files. For details about the files, see .

Table 1 LoRA weight files [object Object][object Object]

[object Object]undefined[object Object]

• The Atlas 800I A2 inference server, Atlas 800I A3 SuperPoD server, and Atlas 300I Duo inference card support this feature.
• The number of LoRA weights is limited by the hardware memory. It is advised to keep the number ≤ 10.
• Dynamic loading and unloading of LoRA weights are supported only when ATB Models use Python to build graphs.
• LoRA weights can be carried by linear layers.
• This feature cannot be enabled together with the quantization, prefill-decode disaggregation, parallel decoding, SplitFuse, MTP, asynchronous scheduling, micro batch, or prefix cache features.
• This feature is supported only by Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, Qwen3-32B, Llama 3.1 8B, Llama 3.1 70B, and Qwen2-72B.
• The length of the LoRA weight name cannot exceed 256 characters.
• This feature supports only vLLM, TGI, and vLLM-compatible OpenAI APIs.

[object Object]

describes the serving parameters required for enabling the Multi-LoRA feature.

Table 2 Additional parameters for the Multi-LoRA feature: Parameters in [object Object] [object Object][object Object]

[object Object]undefined[object Object]

• Pure model usage:

  CANN and ATB Models have been installed in the environment. For details, see MindIE Installation Guide.

  [object Object]

  The following uses Llama 3.1 70B as an example. After downloading the base model and LoRA weights, you can run the following command to perform a dialog test. Three requests form a batch for inference. The LoRA weight in each inference request is different. For details about the parameters in the [object Object] script, see .

  Use [object Object] to specify the binding relationship between the base model and the LoRA weights.

  • The weight name is a weight alias, containing a maximum of 256 characters. It is used to specify the LoRA weight for inference in subsequent requests.

  • Multiple LoRA weights can be configured.

    [object Object]

• Serving usage:

  [object Object]

  The following part uses Llama 3.1 70B as an example to describe how to use Multi-LoRA.

  1. Open the [object Object] file of the server.

     • Installation using the [object Object] package:
     [object Object]
     • Installation using the [object Object] package:
     [object Object]

  2. Set serving parameters. Add the [object Object], [object Object], and [object Object] fields to the [object Object] file of the server. For details about the parameter fields, see . For details about the serving parameters, see . The following is an example of parameter configuration.

     [object Object]

  3. Start the service.

     • Installation using the [object Object] package:
     [object Object]
     • Installation using the [object Object] package:
     [object Object]

  4. Dynamically load, unload, or query LoRA.

     Loading request:

     [object Object]

     Unloading request:

     [object Object]

     Querying request:

     [object Object]

  5. Send a request.

     [object Object] can be set to the base model name (value of [object Object] under the [object Object] field in the [object Object] file) or the LoRA ID (value of [object Object] under the [object Object] field in the [object Object] file). If [object Object] is set to the base model name, no LoRA weights are used for inference. If [object Object] is set to the LoRA ID, the base model weights and the specified LoRA weights are used for inference.

     [object Object]