MindStudio Debugger (msDebug for short) is an operator debugging tool for Ascend devices. It is used to debug operator programs running on NPUs and provides debugging methods for operator developers. The debugging methods include reading the memory and register of an Ascend device, and pausing and resuming the running status of a program. After testing the operator functions in a real-world hardware environment by starting operators or using the msOpST tool, you can determine whether to use the msDebug tool for function debugging based on the actual test situation.
Scenarios
The following operator call scenarios are supported:
Kernel launch operator development: kernel launch
For details about the kernel launch scenario, see section "" in Ascend C Operator Development Guide. For details about the operation, see "".
Project-based operator development: single-operator API calling
For details about the single-operator API execution scenario, see "Project-based Operator Development" > "" in Ascend C Operator Development Guide. For details about the operation, see "".
AI framework operator adaptation: PyTorch framework
For details about the single-operator calling scenario through the PyTorch framework, see "OpPlugin in " in Ascend Extension for PyTorch Suite and Third-party Library Support List. For details about the operation, see "".
Additional Information
msDebug also provides the following extension program. For details, see .
Table 1 Extension program [object Object][object Object]
[object Object]Environment Setup
- Install msDebug by referring to .
- To enable msDebug, install the NPU driver and firmware using either of the following methods (method 1 is recommended for CANN 8.1.RC1 and later, and driver 25.0.RC1 and later):
Method 1: Specify the
[object Object]option during driver installation, and then run the[object Object]command as the[object Object]user to enable the debugging channel. Then the msDebug tool can be used.[object Object]Method 2: Specify the
[object Object]option during driver installation. For details, see "" in CANN Software Installation Guide.[object Object]
Constraints
- The debugging channel has high permissions, which causes security risks. Exercise caution when using this tool. This tool is not recommended in the production environment. If you use this tool, you implicitly accept the risks involved.
- For a single device, only one msDebug tool can be used for debugging. You are not advised to run other operator programs at the same time.
- When the program to be debugged calls multiple operators, the msDebug tool can debug only a specified operator.
- During operator debugging, the overflow/underflow detection function is disabled.
The following products are supported:
- Atlas A3 training products/Atlas A3 inference products
- Atlas A2 training products/Atlas A2 inference products
[object Object]
- You can run the
[object Object]command to view all the commands supported by msDebug. Commands excluded in are implemented by the open-source debugger LLDB. Pay attention to related risks when using LLDB. For details about how to use LLDB, see its . - You need to ensure the execution security of executable files or applications.
- You are advised to restrict the operation permission on executable files or applications to avoid privilege escalation risks.
- Avoid high-risk operations (such as deleting files, deleting directories, changing passwords, and running privilege escalation commands) to prevent security risks.
Table 1 Command reference
[object Object][object Object]
Importing Debugging Information
Before debugging an operator, enable the debugging [object Object] option and recompile the operator to include debugging information in the operator binary. For details, see . The operator debugging information is automatically imported to the msDebug tool.
Starting the tool
The msDebug tool can be started in either of the following ways.
[object Object]
Load the executable file
[object Object].After the operator is built, the executable file
[object Object]on the NPU can be obtained.Use msDebug to load the executable file.
[object Object][object Object]
Load the Python script for operator calling.
After plugins of the PyTorch framework are developed, you can directly call Ascend C custom operators from PyTorch through the custom Python script
[object Object].Use msDebug to load the Python script.
[object Object][object Object]
Exiting Debugging
Exit the debugger.
[object Object]
Specifying a Device ID (MC2 Operator Scenario)
When debugging a single-process multi-thread MC2 operator, you can run the [object Object] command ([object Object] indicates the device ID) to specify the device ID to debug the operator on a specific device. This debugging mode has the following advantages:
- Higher debugging efficiency: By selecting a specific device, you can use hardware resources more efficiently and accelerate the debugging process.
- Well targeted: You can debug a specific device to detect and resolve performance bottlenecks or compatibility issues related to the device.
- Issue isolation: If a performance or function issue occurs, you can specify different device IDs to check whether the issue is caused by a specific device, thereby making it easier to locate the issue.
[object Object]
When using msDebug to debug an operator, you can set line breakpoints on the execution program of the operator, that is, set breakpoints at a specific line in the operator code file.
If an operator implementation file with the same name exists on both the host and kernel, you are advised to use an absolute path to set a breakpoint to ensure that the breakpoint is set on the target file.
When a breakpoint is set on the source code file, an alarm indicating that the actual location cannot be found may be displayed, as shown in the following. After the operator is executed, the actual location is automatically found and the breakpoint is automatically set.
[object Object]If the operator code is compiled into the dynamic library and loaded by using the operator launch symbol, when a breakpoint is set before the
[object Object]command is executed, the command output indicates that the breakpoint position is not found (pending on future shared library load). The dynamic library is loaded only after the program is executed. The operator debugging information is parsed after the[object Object]command is executed, and then the breakpoint is updated and reset.[object Object]
Setting a Line Breakpoint
Add a breakpoint in line 114 of the kernel function implementation file
[object Object]. If the following information is displayed, the breakpoint is successfully added:[object Object]For details about the command output, see the following table.
Table 1 Information description
[object Object]Run the operator program and wait until the breakpoint is hit.
[object Object]indicates the address of the PC where the breakpoint is located.[object Object]
Printing Breakpoints
Run the following command to print the positions and sequence numbers of all breakpoints that have been set.
Deleting Breakpoints
Delete the breakpoint with a specific line number.
[object Object]Resume the running of the program. Due to breakpoint deletion, the program keeps running to the last minute.
[object Object]
Based on the variable type and usage, a variable can be stored in a register or in the local memory or global memory. You can determine the storage location by printing the variable address and further view the associated memory content.
Currently, the msDebug tool cannot directly print the value of a template parameter by variable name. You need to print the value of the template parameter using the [object Object]. The value of the template parameter is displayed after printing. For example, [object Object] is a template parameter, and [object Object] is the object pointer to which the template parameter belongs. If you want to print the value of the parameter, run the [object Object] command where the parameter is used. An example is provided as follows:
Printing Variables
After a breakpoint is hit, you can run the [object Object] command to print the value of a specified variable. For example:
Printing GlobalTensor
[object Object] is used to store the global data of the global memory (external storage).
You can run the following commands to print [object Object]. The following takes [object Object] as an example. The [object Object] field specifies the memory address of [object Object]. In this example, the value is [object Object].
The actual values of [object Object] variables are stored in the GM. Run the following command to print the values at [object Object] in the GM. The example printing format contains the following information: one line to be printed, 256 bytes in each line, in float32 format.
[object Object]
Printing LocalTensor
[object Object] is used to store the data in the local memory (internal storage) of the AI Core.
Run the following command to print the [object Object] variable. [object Object] is used as an example. For the memory address of [object Object], refer to the [object Object] parameter in the [object Object] field. In this example, the address is [object Object], and the length is [object Object].
The actual content of the tensor is stored in the UB memory. You can run the following command to print the value at address [object Object] in the UB memory. The example printing format contains the following information: one line to be printed, 256 bytes in each line, in float32 format.
[object Object]
Printing All Local Variables
Print all local variables in the current scope:
To understand the code execution details, you can run the [object Object] command to execute the code line by line for single-step debugging, or run the [object Object] command to enter the function for debugging, or run the [object Object] command to return to the next line of the function call point to continue debugging.
During operator build, the build option of [object Object] is used.
Example for Using the thread step-over Command
Set a breakpoint to the position to be debugged and run the program. For details about how to set a breakpoint, see .
[object Object]Enter the
[object Object]or[object Object]command for step-by-step execution.[object Object]Run the
[object Object]command to view the PC information and stop reason of all cores.[object Object][object Object]
After the debugging is complete, run the
[object Object]command and enter[object Object]or[object Object]to end the debugging.[object Object]
Example for Using the thread step-in and thread step-out Commands
Set a breakpoint to the position to be debugged and run the program. For details about how to set a breakpoint, see .
[object Object]Input
[object Object]or[object Object]to enter the function for execution.[object Object]Run the
[object Object]command to view the PC information and stop reason of all cores.[object Object][object Object]
After debugging the CopyOut function, run the
[object Object]command to exit the CopyOut function and return to the main program to continue execution.[object Object]
When the operator execution program freezes, manually interrupt the operator execution program and display the interrupted location information.
If the running program freezes, you can press "Ctrl+C" to interrupt the program. The possible causes of freezing are as follows:
- The user program itself has an infinite loop, which needs to be rectified by repairing the program.
- An operator uses synchronization instructions.
This function can debug only the operator programs started in msDebug.
After the interruption takes effect, the debugging information displaying and core switching functions are supported. Currently, single-step debugging, register reading, memory and variable printing, and
[object Object]command are not supported.
When the operator execution program on the host or device suspends, enter "CTRL+C" to manually interrupt the operator execution program and display the interrupted location.
[object Object]After the debugging is complete, run the
[object Object]command and enter[object Object]or[object Object]to end the debugging.[object Object]
Switch the current core to the specified core. After the core is switched, the position of the code interruption of the specified core is automatically displayed.
Assume that the running core is core 2 of the AIV, and the core to be switched is core 3 of the AIV.
[object Object]After the switchover is complete, query the core information again. You can see that the core is switched to the line where the new core ID is located.
[object Object]Assume that the running core is core 3 of the AIV, and the core to be switched is core 17 of the AIC.
[object Object]After the switchover is complete, query the core information again. You can see that the core is switched to the line where the new core ID is located.
[object Object]
After using msDebug to call an operator, you can read register values of the device where the current breakpoint is located to check the program status.
After
[object Object]is entered, all available register values on the current device are returned.[object Object]After
[object Object]is entered, the register value on the current device is returned. Separate multiple registers with spaces.- The register value is returned when the variable name is available on the current device.
[object Object]is returned when the variable name is not available on the current device.
[object Object]
Query information about the device where the operator runs.
ascend info devices
Run the following command to query the information about the device where the operator is running. The line where [object Object] is located indicates the target device.
[object Object]
For details about the command output, see the following table.
Table 1 Information description
[object Object]ascend info cores
Run the following command to query the information about the core where the operator is running. The line where [object Object] is located indicates the target core. In the following example, the target core is core 0 of the AIV.
For details about the command output, see the following table.
Table 2 Information description
[object Object]ascend info tasks
Run the following command to query the task information of the operator. The line where [object Object] is located indicates the target task, including device ID, stream ID, task ID, and invocation (name of the called kernel function).
ascend info stream
Run the following command to query the stream information of the operator. The line where [object Object] is located indicates the target stream, including device ID, stream ID, and type (kernel type, which can be AIC or AIV).
ascend info blocks
Run the following command to query the block information of the operator. The line where [object Object] is located indicates the target block, including device ID, stream ID, task ID, and block ID.
Run the following command to print the code of the running block at the current breakpoint:
If a hardware issue happens onsite, repeated stress tests are needed to reproduce the issue, which slows down troubleshooting. To solve this problem, the system initiates a dump operation upon detecting a potential hardware issue, and captures the current status information. The msDebug tool parses the dump file of an abnormal operator. You can collect sufficient data for fault analysis even without a stress test. The above functions enhance hardware exception detection and minimize repetitive stress tests.
[object Object]
After the [object Object] file is configured, other functions of msDebug cannot be used.
Prepare the
[object Object]configuration file.- Project-based operator development (single-operator API calling scenario): Create the
[object Object]file by referring to "" in Application Development Guide (C&C++) and load the file using the[object Object]API. - AI framework operator adaptation (PyTorch framework scenario): Search for the
[object Object]file in the installation directory of[object Object].
- Project-based operator development (single-operator API calling scenario): Create the
Enable the function of generating dump files for abnormal operators by referring to the configuration file example (dump configuration for abnormal operators) in "acl API Reference (C)" > "System Configuration" > "" in Application Development Guide (C&C++).
- In the
[object Object]configuration file, set[object Object]to[object Object]. - In the
[object Object]configuration file, set[object Object]to the path for exporting the dump file of the abnormal operator.
- In the
If the program crashes (for example, memory overflow or segmentation fault), a core file of the abnormal operator is generated. The file name ends with .core.
Run the following command with the msDebug tool to load the dump file of the abnormal operator:
[object Object][object Object]
View the dump file information of the abnormal operator.
[object Object]For details about how to locate hardware exceptions, see , , and .
After the debugging is complete, run the
[object Object]command and enter[object Object]or[object Object]to end the debugging.[object Object]