Ensuring inference accuracy for large language models (LLMs) on Ascend NPUs aims to achieve semantic equivalence with their GPU-based counterparts. This is typically assessed through downstream task evaluation or direct comparison with GPU outputs. In practice, accuracy issues often stem from model structure errors, improper parameter configurations, or operator calculation deviations. These issues typically manifest as abnormal behaviors such as incoherent output (gibberish), unexpected generation interruption, repetitive generations, or significant deviation from expected results. Accuracy issues in model inference can be classified into two categories: model accuracy issues and numerical precision issues. These two types differ in their causes and impacts, and therefore require differentiated analysis and handling strategies.
Model accuracy issues are "structural deviations". They include data loading exceptions, incorrect configuration of training hyperparameters, incorrect implementation of network structures, or design defects within the framework itself. Such issues have a decisive impact on model convergence. Consequently, you must carefully examine each phase and make targeted adjustments based on the specific scenario.
Numerical precision issues are "computational deviations", arising from the inherent characteristics of floating-point operations. Limited word length prevents full representation of the value range, and differences in computation and communication order, as well as the approximation of mathematical expressions, can all introduce errors. It should be noted that although approximations in numerical computation may interfere with the convergence process, computational discrepancies do not necessarily lead to convergence failure. As the basic unit of computation, an operator's numerical precision is a key consideration. However, due to differences in hardware architecture (e.g., between GPU and CPU, or across different GPU versions), slight numerical deviations in the same computational logic are normal. As long as these deviations fall within a reasonable tolerance, they will not affect the final convergence of your model.
This guide is designed to help you effectively distinguish between normal computational deviations and genuine accuracy issues, and to accurately locate their root causes. To achieve this, it systematically describes the application scenarios and operational workflows of a dedicated toolset for accuracy issue diagnosis. The goal is to empower you to efficiently identify potential risks, either independently or with the support of technical documentation.
Accuracy issues in LLM inference typically manifest as model outputs that deviate from expectations. These symptoms can be classified into the following types:
- Garbled characters in model responses: A large number of abnormal symbols such as �, <unk>, and ’ are displayed in outputs, or a segment of characters in another language is suddenly inserted.
- Repeated model responses: A model is stuck in a certain part and repeatedly generates the same or similar text segments.
- Semantic or logical errors: The output uses fluent language but the reasoning is either incoherent or broken.
- Inconsistency and jitter: The outputs vary greatly among multiple requests.
- Dataset evaluation discrepancy: The evaluation accuracy falls below the benchmark.
Common accuracy issues in LLM inference can be classified into accuracy errors and practice errors. In inference deployment, practice errors are the primary cause of accuracy issues. The causes of practice errors are as follows:
(1) Model configuration error
Includes abnormal weights and incorrect model parameter configuration.
(a) Incorrect weights can affect the word embedding table, linear, layerNorm, and lmhead.
(b) Parameter configuration includes the padding mode and model [object Object] settings, such as [object Object], [object Object], and [object Object]. You can avoid errors by aligning [object Object].
(2) Model structure error
Includes code implementation error. Typically, this leads to obvious accuracy errors, manifesting as disordered or nonsensical English output, or even no output at all.
(3) Operator parameter pass error
(a) The attention mask is incorrectly passed. Generally, a model does not output disordered characters. However, there are cases where, while the model can answer questions, it may insert an excessive number of spaces. Consequently, the answers may not align with those produced on the GPU
(b) The RoPE rotation coefficient is incorrectly passed. Consequently, a model usually outputs disordered results.
(4) Operator implementation error
(a) Operator implementation postprocessing: Generally, the accuracy is normal during greedy search of a model, but the output is incorrect after the postprocessing sampling policy is added. For example, the output is repeated or similar to the greedy search result.
b. Model-sided operator: Single-operator detection passes but the overall model behaves abnormally. Potential causes may include memory corruption caused by interaction among multiple operators, and improper resetting of registers. You can ask the same question for multiple times after deterministic computing is enabled. If the results are inconsistent, the operator implementation may be incorrect.
(5) Environment version defects or differences
Generally, accuracy may be normal on one type of server but suddenly become abnormal on other servers or in other environments, or after an environment version change. For example, accuracy may become abnormal when an x86 architecture is switched to an Arm architecture, or after CANN is replaced. For the solutions, see the requirements of different environment versions.
The following figure shows the overall process of locating model inference accuracy issues.
In scenarios where a benchmark is available, it is critical to ensure that your configuration aligns with the benchmark setup. For model inference that involves benchmarking, two specific scenarios are typically considered:
- NPU inference versus GPU inference of the same model, with GPU serving as the benchmark.
- NPU inference of the current model versus that of a historical model version, with the historical baseline serving as the benchmark.
In the following description, these two scenarios are essentially a comparison between the problem scenario and the benchmark scenario. Pay attention to the following items in the checklist:
- Comparison of inference hyperparameters and environment variables
You can use Beyond Compare to compare inference hyperparameters and environment variable settings in the inference logs or startup scripts in both scenarios.
- Comparison of third-party library versions
Check whether the versions of third-party libraries such as MindIE, vLLM, and Transformers are consistent with those in the benchmark version by [object Object].
Check whether the versions of third-party libraries such as PyTorch and PTA are consistent with those in the benchmark version by [object Object].
- Data reading check
Check the input data of model inference. Generally, you can directly print the input data in the code for checking. For example, in vLLM, you can obtain the original input text through [object Object] in the output of [object Object].
- Model configuration check
You can directly print model configurations of both scenarios for comparison.
Example
Output example:
After ensuring that the preceding information is consistent, you need to reproduce a problem. To minimize variables during fault localization, you should fix randomness and enable operator determinism.
To reproduce a problem, you need to fix the random operations to ensure experimental reproducibility. These operations include model parameter initialization and dropout layers. The involved operations are as follows:
- Fix the random seed, such as
[object Object],[object Object], and[object Object]. - Disable the dropout layer.
During problem reproduction, you are advised to enable operator computation determinism and communication determinism. Both of them need to be fixed as early as possible before the code starts training.
- Computation determinism:
[object Object] - Communication determinism:
[object Object]
To achieve the randomness fixing and determinism enabling operations described above, the msProbe toolkit provides the [object Object] API. In PyTorch scenarios, this API can quickly fix all random seeds, dropout layers, and ensure deterministic operator computation and communication within the network.
How to use:
Parameter description:
In model inference accuracy localization, two different models may exhibit different performance on the same dataset. For example, in a problem scenario, a model may answer a question incorrectly on a dataset evaluation, whereas the benchmark scenario produces the correct answer. Such an instance is referred to as a bad case.
Once a bad case is identified, subsequent issues can be localized as single-case problems. The subsequent operations will be described in the following section.
In model inference, reproducible single-case accuracy issues typically refer to an extremely stable bad case. The details are as follows:
Fixed input: A specific prompt is given (for example, "What is the capital of China?").
Fixed parameters: [object Object], [object Object], fixed [object Object] fixed, and consistent [object Object]
Incorrect output: For example, the model consistently answers "Shanghai" instead of the correct answer, "Beijing".
vLLM is a high-performance model inference framework developed by the team at UC Berkeley. It uses innovative memory management and scheduling policies to address the problems of low memory utilization, insufficient throughput, and low concurrent processing efficiency in traditional inference frameworks when deploying foundation models. The core advantages of vLLM lie in its unique memory management mechanism (PagedAttention) and Continuous Batching technology. These two innovations enable memory utilization to reach nearly 100% and the throughput to be 24 times that of traditional frameworks, making vLLM particularly suitable for real-time inference scenarios with high concurrency and low latency. The inference process of vLLM is divided into two main phases: prefill and decode. The prefill phase processes input prompts and generates the initial KV Cache, while the decode phase generates output tokens one by one and continuously updates the KV Cache. Throughout the process, vLLM uses its PagedAttention and Continuous Batching features to efficiently utilize memory resources and fully schedule computing resources.
To locate accuracy issues in vLLM scenarios, the dump and comparison capabilities of the msProbe tool are mainly used. vLLM involves multiple startup modes. The following describes how to enable the tool in each startup mode, using vLLM 0.9 as an example.
- V0, offline mode, TP = 1
Obtain the model:
[object Object]
For details about the configuration file, see . For details about msProbe interfaces, see documents related to precision data collection in torch scenarios. You can set the [object Object] parameter in the [object Object] interface to control the token data to be collected.
- V0, offline mode, TP > 1
The multiprocessing executor [object Object] is used, which introduces a process interval. In this setup, rank 0 resides in the main process, while all other ranks are in subprocesses.
Tool adding position: For data collection on rank 0, the tool can be directly added to the outermost layer of the [object Object] function called by LLM. For data collection on other ranks, the tool needs to be added to the [object Object] function of the subprocess ([object Object]).
- V0, online mode, PP = 1 (TP and DP not limited;
[object Object]not set)
The multi-process client [object Object] is used, causing a process interval. All ranks are in subprocesses.
Tool adding position: [object Object] function of the [object Object] class in the subprocess ([object Object])
- V0, online mode, PP > 1 or
[object Object]specified
The multiprocessing executor [object Object] is used, which introduces a process interval. In this setup, rank 0 resides in the main process, while all other ranks are in subprocesses.
Tool adding position: See "V0, online, TP > 1" mode.
[object Object],[object Object]
NPU: [object Object] function in [object Object]:
GPU: [object Object] function in [object Object]
Add the corresponding code according to the configuration (L0/L1).
NPU: [object Object] function in [object Object]:
Disable the function (place the code before the [object Object] statement in the [object Object] function):
GPU->vllm/v1/worker/gpu_model_runner.py GPUModelRunner.execute_model
Generally, there are three phases for locating accuracy problems that can be reproduced in a single case.
The accuracy benchmark may come from either a GPU or a historical version of the NPU baseline that is known to have normal accuracy.
For details about model configuration check and randomness fixing, refer to and . In the vLLM scenario, you also need to fix sampling randomness ([object Object] = [object Object]).
- Confirming the first different token
You can print the output token ID sequence of the test case in vLLM. The following uses the V1 scenario as an example.
Then, you can easily compare the position of the first different token in the problem scenario with that in the benchmark scenario.
- Using msProbe to dump data
For details about how to enable the msProbe dump feature, see . The mix level and statistics mode are recommended for data dump. The data in the following format can be obtained.
After the data dump, two files are obtained, one for the problem scenario and the other for the benchmark scenario.
You can use to compare the data. The following is an example:
[object Object]
After the comparison is complete, a comparison table is generated. In the table, you can view the differences ([object Object]) of four statistics. By examining the larger [object Object], you can locate the suspicious operator.
As shown in the preceding figure, matmul is the suspicious operator. You can reproduce the problem on this operator.
Mind Inference Engine (MindIE) is an inference acceleration suite provided by Ascend for various AI scenarios. Through layered open AI capabilities, it supports diverse AI service needs and empowers a large number of models by leveraging the compute of Ascend hardware. MindIE supports multiple mainstream AI frameworks and is compatible with different types of Ascend AI processors, providing multi-layer programming APIs to help users quickly build inference services based on the Ascend platform.
Currently, MindIE is often used together with the acceleration library to achieve optimal inference performance. The following uses MindIE + ATB as an example to describe how to locate accuracy problems in MindIE scenarios.
The following describes how to use the dump and comparison functions of the msProbe tool to locate accuracy problems in the MindIE scenario.
Tool Installation
Currently, the WHL msProbe installation package capable of dumping accuracy data for the ATB model can only be obtained by compiling the source code. The compilation and installation procedure is as follows:
Notes:
(1) The third-party dependencies such as Git, curl, GCC 7.5 or later, and CMake 3.19.3 or later must be installed in the compilation environment.
(2) If the compilation fails due to a security certificate issue, you can temporarily disable the security certificate verification if the environment is secure. The compile command for disabling certificate verification is [object Object].
Usage
Create a
[object Object]file to set dump parameters. Example:[object Object]Parameters in the Dump Configuration File
The dump configuration file is a text file in JSON format. The table below describes its parameters.
[object Object]undefined
Run the script for loading the ATB dump module. Example commands:
[object Object][object Object]indicates the msProbe installation path,[object Object]indicates the output path of the dump data, and[object Object]indicates the path of the dump configuration file.Perform an MindIE inference job.
msProbe automatically collects the accuracy data during ATB model running. After obtaining the dump data, you can use the comparison function of msProbe to compare and analyze the dump data. The comparison command is as follows:
[object Object]Comparison command parameters
[object Object]undefined
For details about how to use the dump and comparison functions, see the and .
If a precision issue cannot be located, you can narrow down the scope hierarchically: first identify the problematic layer, and then locate the specific op within that layer. Here, an op includes both operation and kernel under the layer.
Layer-level Problem Locating
Set the [object Object] parameter in the dump configuration file to [object Object] and [object Object] to [object Object] to dump only the input and output tensors of all layers. The following is an example of the dump output.
Compare the dump data between a layer with a precision issue and a layer without any precision issue. Then, identify the earliest layer where the precision issue first appears. The following is an example of the comparison result.
As shown in the preceding figure, the first layer where a precision problem occurs is [object Object].
OP-level Problem Locating
In the dump configuration file, set the [object Object] parameter to the ID of the layer where a precision issue occurs (e.g., [object Object]) and set [object Object] to [object Object] to dump the input and output tensors of all operations and kernels under the specified layer. The following is an example of the dump output.
Compare op-level dump data from a normal run and a problematic run to pinpoint the operator likely causing the precision issue. The following is an example of the comparison result.
As shown in the preceding figure, the [object Object] operator may have a precision issue.
After locating the suspicious operator, you can use the dump data to reproduce the problem in single-operator scenario. This helps to further determine whether the precision issue originates from the operator itself or from other factors, such as memory corruption.