Operator fusion, quantization, tensor parallelism, and continuous batching.
During pure model inference, the error message "out of memory, need block" is displayed, as shown in the following figure.
This typically occurs when sequence length increases due to large images or videos, making the pre-allocated KV cache insufficient.
Change the value of [object Object] in the [object Object] script to a larger value based on the actual application scenario.
There are a large number of experts on a single device, and each expert is allocated with a small number of tokens. When the precision is different, the performance of a chat interface may be worse than that of a non-chat interface.
The experts activated by the chat interface are more evenly distributed, but the number of experts activated by a single device is larger. As a result, more experts need to be transferred, the performance deteriorates, and the GMM operator performance fluctuates.
This is an inherent difference between chat and non-chat interfaces, which is normal.
Check whether MindIE LLM matches ATB, CANN, torch, and torch_npu, and whether the value of [object Object] ([object Object] or [object Object]) is correctly selected.
When torch.distributed is enabled for multi-device inference, the [object Object] or [object Object] environment variable does not exist on the server.
The environment variable the [object Object] or [object Object] environment variable is not set.
You can set the environment variable in either of the following ways:
Setting in the code
[object Object]Using environment variables
[object Object]
The error message "Max retries exceeded with url" is displayed during model startup, as shown in the following figure.
This issue is most likely caused by intranet access.
The solution uses Qwen-VL as an example. Open the [object Object] file in the weight folder and modify the 29th and 30th lines as follows.
After a model is loaded, the program exits quickly, and the error message "Socket bind failed" is displayed, as illustrated in the following figure.
MindIE Motor uses HTTP or HTTPS for communication. The client can disconnect first to reduce the server load and ensure proper release of resources, including port resources.
Modify the [object Object], [object Object], and [object Object] parameters in the configuration file. To avoid this problem, disconnect requests before disconnecting the service process. Alternatively, run the [object Object] command to check the port status. If the port is still occupied by a residual process, run the [object Object] command to clear the residual process. Replace [object Object] with the port number to be checked.
After serving is started successfully, no response is returned after a request is sent.
Check model logs to see if "out of memory" occurs.
Change the values of [object Object] and [object Object] in the configuration file.
If serving fails to be started, check logs, which are stored in [object Object] by default.
Run the [object Object] command to upgrade ProtoBuf.
During serving startup, the error message "Check_path: config.json failed" is displayed, as shown in the following figure.
The [object Object] file in the model weight path does not have the 640 permission.
Use either of the following methods to change the permission:
Change the permission on the
[object Object]file.[object Object]Change the permission on the entire model weight folder.
[object Object]
The error "pybind11::error_already_set" is reported during service startup, as illustrated in the following figure.
The model's third-party dependencies are incorrect.
Reinstall the third-party dependencies based on the [object Object] file of the model. The default path of the dependency file is [object Object].
No core dump log is generated during service startup.
The model's third-party dependencies, such as Protobuf, are incorrect.
Reinstall the third-party dependencies based on the [object Object] file of the model. The default path of the dependency file is [object Object].
After the environment variable is enabled, run the model inference and further locate the fault based on the first error in the acceleration library log.
Deterministic computing refers to the process of running an inference application multiple times with unchanged inputs, such as an input dataset, ensuring that the output results are consistent each time.
- After the model postprocessing is changed from sampling to greedy, the stability of the output text can be ensured.
- Due to the problem of deterministic computing, the output may be slightly different.
- The accumulation sequence of the MatMul operator varies across different rows. Additionally, floating-point precision lacks the commutative property of addition. Consequently, even if the input is the same across different rows, the calculation results are different.
- You can set the environment variable
[object Object]to[object Object]to disable the shuffle k function of MatMul. After it is disabled, the operator accumulation sequence on all rows can be the same. However, the performance of MatMul decreases by about 10%.
The scheduling framework code (in block query mode) can ensure deterministic scheduling. However, environmental factors such as CPU load can affect the request arrival time, ultimately impacting deterministic scheduling. For example, after an engine query, 10 requests can be submitted for a customer's external services. During the first run, all 10 requests quickly arrive and form a batch. However, during the second run, some requests arrive late due to environmental factors, and only five requests form a batch. As a result, the results of the two runs are different.
An error associated with fault location occurs during pure model inference.
Model inference involves asynchronous execution, which may result in misleading error messages. Ensure synchronization before performing fault location.
Set [object Object] to [object Object] to enable synchronous execution before fault location.
Deterministic computing refers to the process of running an inference application multiple times with unchanged inputs, such as an input dataset, ensuring that the output results are consistent each time.
Model level:
Communication operator:
[object Object]MatMul:
[object Object]Inference engine:
MindIE: Obtains new requests based on blocks.
TGI: not supported.
Check whether the tokenizer uses the correct model path when converting a token to an ID.
During inference, the "Pull kv failed" log of the ERROR level is generated when the decode node pulls the KV cache, and the error code "timeout" is displayed in [object Object] of CANN.
In the prefill-decode disaggregation scenario, the KV cache of the decode node needs to be pulled from the prefill node. If this error occurs, the KV cache transmission from the prefill node to the decode node times out, which is probably caused by poor network quality.
(Recommended) Check the number of network transmission retries. If certain devices exhibit an unusually high retry count, inspect the corresponding optical modules.
[object Object]Set
[object Object]to[object Object]in the[object Object]field of the MindIE configuration file, to specify the timeout interval for pulling the KV cache to 5 seconds. However, this setting may mask inference performance degradation caused by network problems. Exercise caution when configuring it.
During MindIE LLM deployment, the error message "LLMPythonModel initializes fail" is reported, as shown in the following figure.
There is no Python dependency for ibis.
Go to the [object Object] directory, open Python logs, and install the required dependencies based on the error information in logs.
During MindIE LLM deployment, the error message "out of memory" is displayed when the model is loaded, as shown in the following figure.
The weight is too large and the memory is insufficient.
Set [object Object] of [object Object] in the [object Object] file to a smaller value, for example, [object Object].
During MindIE LLM deployment, [object Object] cannot be imported, as shown in the following figure.
The Python version is not 3.10, or the Python version corresponding to pip is not 3.10. As a result, the corresponding package cannot be found. You can run the [object Object] and [object Object] commands to check the Python version.
Open the
[object Object]file.[object Object]Add the following environment variables to the
[object Object]file, save the file, and exit.[object Object]Make the environment variables take effect.
[object Object]Create soft links.
[object Object]
During MindIE LLM deployment, paths such as [object Object] cannot be found, as shown in the following figure.
When the HTTPS service is enabled, the required certificates are not stored in the corresponding directories.
Save the files required for authentication, such as the server certificate, CA certificate, and server private key, to the corresponding directories.
The following figure displays the error details.
The [object Object] file has a bug.
Run the following command:
It takes about 3 hours to load a 1300B model. "B" stands for "Billion".
Asynchronous loading is not used.
Set the environment variable [object Object] to optimize model loading. [object Object] is used to set the number of threads of the Open Multi-Processing (OpenMP) parallel programming framework. After the setting, it takes about 10 minutes to load a 1300B model.
In addition, run the following command to start collecting NPU graphics memory fragments:
When the multimodal model input ([object Object]/[object Object]/[object Object]) is used, the following error message is displayed:
OpenAI API:
[object Object]vLLM API:
[object Object]Triton API:
[object Object]
The input image, audio, or video is encoded using Base64 (the data after Base64 encoding is usually 4/3 times that of the original data). As a result, the entire [object Object] exceeds 4MB, and an error is reported.
Method 1: Referring to the API restrictions
OpenAI API:
The total size of all fields in the messages parameter in the request cannot exceed 4MB. For details, see inference APIs.
vLLM API:
The total size of all fields under the
[object Object]parameter in the request cannot exceed 4MB. For details, see "API Reference" > "RESTful API Reference" > "EndPoint Service Plane RESTful APIs" > "Compatible with vLLM 0.6.4 APIs" > "Text/Streaming Inference APIs" in MindIE LLM Development Guide.Triton API:
The total size of all fields under the
[object Object]parameter in the request cannot exceed 4MB. For details, see "API Description" > "RESTful API Reference" > "EndPoint Service Plane RESTful APIs" > "Compatible with Triton APIs" > "Text Inference APIs" in MindIE LLM Development Guide.[object Object]
Method 2: Manually modifying the source code
For example, change the upper limit of
[object Object]to 10MB by modifying the code as follows:Figure 1 Example 1
Figure 2 Example 2
When running inference with the multimodal model, an error similar to the following occurs in [object Object]:
Figure 1 Error message
Figure 2 Error file
Figure 3 Error file
The issue may stem from a [object Object]-related operator where tensor 5 has size 1 in a certain dimension, but the required size is 2, causing a mismatch. This may be related to [object Object], as it removes dimensions of size [object Object].
Example: If an operator (such as concat or MatMul) expects a dimension to exist and match a specific value (for example, 2), an error will occur if that dimension has been removed by a squeeze operation.
[object Object][object Object]
Modify the code as follows:
Running Qwen2.5-VL series models fails, and an error message similar to either of the following is displayed:
Error message 1:
[object Object]Error message 2:
[object Object]
The model configuration is not supported because the installed dependencies are incorrect. You need to install the corresponding dependency files.
[object Object]Handling method for error message 1:
Install the corresponding
[object Object]file based on the dependencies required by each model.The path of the common dependency file to be installed for all models is as follows:
[object Object]The dependency file of each model is in the
[object Object]directory. For example, the dependency file of the Qwen2-VL model is as follows:[object Object]The installation command is as follows:
[object Object]
Handling method for error message 2:
Click to check whether the model is supported by the hardware environment.
Run the following command to check whether the driver version is correct. The driver version must be 23.0.7 or later. You are advised to install the driver of 24.1.RC2 or later.
[object Object]Check whether the initial environment variables have been configured and taken effect.
Check whether the free memory of the system is sufficient.
Run the following command to check the free memory size. Ensure that the size is greater than Weight size/Number of machines.
[object Object]Based on experience, try to keep
[object Object].[object Object]
Import the following environment variables:
[object Object]Check whether the multi-node serving parameter settings are consistent.
Restart the server and restart the service.
[object Object]
During pure model inference and serving startup/inference, various out of memory (OOM) errors are reported. The error information is similar to the following:
- The model weight file is large.
- The input shape is too large (the batch size is large, the text is too long, or the image, audio, or video is too large).
- The values of some parameters in the configuration file are too large.
Increase the value of the
[object Object]environment variable (indicating the memory allocation ratio; default value[object Object]). The following is an example:[object Object]Lower the values of parameters such as
[object Object],[object Object],[object Object],[object Object], and[object Object]in the serving configuration file[object Object]. Focus primarily on adjusting[object Object], and[object Object].The value of
[object Object]must be greater than or equal to that of[object Object].[object Object]affects the workspace in the ATB initialization phase. If the value is too large, an OOM error may be reported after the service is started.
Adjust the value of
[object Object](the maximum memory a single NPU can allocate for KV cache. The default value is[object Object], indicating that KV cache is automatically allocated. If the value is greater than 0, KV cache is manually allocated and the KV cache size is fixed based on the set value).[object Object]= Total memory of a single NPU x Memory allocation ratioUse more NPUs. For example, if two NPUs are used, you can increase the number of NPUs to four or eight. However, you need to confirm the number of NPUs supported by the current model on the current hardware.
An error message similar to the following is displayed when the multi-modal model is used for inference:
Or
The model may not support the hardware environment in the current version.
The input shape is too large, which is not supported by the self-attention operator.
The model may not support the hardware environment in the current version.
The input shape is too large, which is not supported by the self-attention operator.
Reduce the value of
[object Object]in the serving configuration file[object Object].
An error message similar to the following is displayed when the [object Object], [object Object], or [object Object] format is used for multi-modal model inference:
The value in the [object Object], [object Object], or [object Object] parameter does not meet the specified requirements.
Image
Format 1:
[object Object]–[object Object]supports local paths, Base64-encoded JPG, and HTTP/HTTPS URLs.Format 2:
[object Object]–[object Object]supports local paths, Base64-encoded JPG, and HTTP/HTTPS URLs.Format 3:
[object Object]– local paths only.Format 4:
[object Object]– Base64 encoding only. Supports JPG, JPEG, PNG. MIME types are listed below.
Video
Format 1:
[object Object]–[object Object]supports local paths, HTTP, and HTTPS URLs.Format 2:
[object Object]–[object Object]supports local paths, HTTP, and HTTPS URLs.Format 3:
[object Object]– local paths only.Format 4:
[object Object]– Base64 encoding only. The source format can be MP4, AVI, or WMV. The following table lists the corresponding MIME types. In addition, the length of an encoded video may exceed the maximum length of the MindIE Service request. Therefore, you are advised not to transfer Base64-encoded videos.
Audio
Format 1:
[object Object]–[object Object]supports local paths, HTTP, and HTTPS URLs.Format 2:
[object Object]–[object Object]supports local paths, HTTP, and HTTPS URLs.Format 3:
[object Object]– local paths only.Format 4:
[object Object]– Base64 encoding only. The source format can be MP3, WAV, or FLAC. The following table lists the corresponding MIME types.[object Object]undefined
Format 5:
[object Object]– When[object Object]is[object Object], only Base64-encoded data is supported. The source format can be MP3, WAV, or FLAC, and must be explicitly specified using the[object Object]field.
The tokenizer of Qwen2-VL series models reports an error (other models may also report errors, which are irrelevant to the models). The following error message is displayed:
Figure 1 Error message
- The version of the transformers/tokenizer adapted to the model is incorrect.
- The
[object Object]parameter is set to[object Object]. - The permissions on the serving
[object Object]file, model weight path, and model[object Object]file are incorrect. - The
[object Object]file may be missing from the model weight files. - The vocabulary file is damaged.
The version of the
[object Object]/[object Object]adapted to the model is incorrect.Check the required
[object Object]version for each model, typically listed in the model's[object Object]file. Then, check whether the[object Object]version in the[object Object]file under the model weight path is the same as that in the[object Object]file under the model weight path.Use the following
[object Object]verification method to create a Python script. If the script runs successfully, the[object Object]can be loaded correctly.[object Object]
The value of
[object Object]is[object Object].- Set the
[object Object]parameter to[object Object].
- Set the
The permissions on the serving
[object Object]file, model weight path, and model[object Object]file are incorrect.- Change the permissions on the serving
[object Object]file, model weight path, and model[object Object]file to[object Object].
- Change the permissions on the serving
The
[object Object]file may be missing from the model weight files.- If the
[object Object]file is missing, add it.
- If the
The vocabulary file is damaged.
Run the following command to check the integrity of the
[object Object]file:[object Object]
The following error message is displayed when a Qwen2.5 series model is deployed on MindIE for quantization inference:
Alternatively,
The [object Object] field is not configured.
When performing quantization inference, add the [object Object] field to the [object Object] file in the path where the quantized weights are located. The field value is the quantization mode of the current quantized weights. The following is an example:
When MindIE is used for inference, the model inference output results are different if the input is the same but the batch sequence is different.
[object Object]The accumulation sequence of the MatMul operator varies across different rows. Additionally, floating-point precision lacks the commutative property of addition. Consequently, even if the input is the same across different rows, the calculation results may be inaccurate.
[object Object]Deterministic computing refers to the process of running an inference application multiple times with unchanged inputs, such as an input dataset, ensuring that the output results are consistent each time. You can set the environment variable [object Object] to [object Object] to disable the shuffle k function of MatMul. After it is disabled, the operator accumulation sequence on all rows can be the same. However, the performance of MatMul decreases by about 10%.
Communication operator:
MatMul:
When the MindIE LLM service is started, a Gloo connection error occurs. Information similar to the following is displayed in the log:
This error usually occurs in multi-node deployment scenarios. The core cause is that the Gloo component automatically selects an incorrect network interface (NIC), resulting in a failure in communication between nodes.
Explicitly specify the network interface used by Gloo through environment variables.
View available NICs. Run the following command on each node to view the NIC name:
[object Object]Common NIC naming formats:
[object Object],[object Object]and[object Object]Set environment variables. Before starting the service, set the
[object Object]environment variable for each node.[object Object]Precautions for container deployment:
When using Docker, set the container network mode to
[object Object].The NIC name of each machine may be different. You need to set the NIC name to that of the local host.
Precautions for Kubernetes deployment:
In a Kubernetes cluster, the NIC name is usually mapped to
[object Object].For Moe EP deployment, you can configure environment variables in the
[object Object]script.
After the setting is complete, restart the service and check whether the Gloo connection error is rectified.