Board Debugging on the NPU

Debugging with DumpTensor and printf

The functions of printing board data on the NPU include DumpTensor and printf. DumpTensor is used to print the data of a specified tensor, and printf is used to print scalar and string information.

Usage:

Call the DumpTensor or printf API to print related content where log information needs to be output in the implementation code of the operator kernel.
  • In the following example of DumpTensor, srcLocal indicates the tensor to be printed, 5 indicates the additional custom information, such as the current code line number, and dataLen indicates the number of elements. For details about the usage and restrictions of the DumpTensor API, see DumpTensor.
    1
    DumpTensor(srcLocal,5, dataLen);
    

    During dump, the corresponding DumpHead (32 bytes) is added before the dump information of each block core to record the core ID and resource usage. DumpTensorHead (32 bytes) is also added before the tensor data to be dumped each time to record tensor information. An example of the printing result is as follows:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    DumpTensor: desc=5, addr=0, data_type=float16, position=UB, dump_size=32
    [19.000000, 4.000000, 38.000000, 50.000000, 39.000000, 67.000000, 84.000000, 98.000000, 21.000000, 36.000000, 18.000000, 46.000000, 10.000000, 92.000000, 26.000000, 38.000000, 39.000000, 9.000000, 82.000000, 37.000000, 35.000000, 65.000000, 97.000000, 59.000000, 89.000000, 63.000000, 70.000000, 57.000000, 35.000000, 3.000000, 16.000000,
    42.000000]
    DumpTensor: desc=5, addr=100, data_type=float16, position=UB, dump_size=32
    [6.000000, 34.000000, 52.000000, 38.000000, 73.000000, 38.000000, 35.000000, 14.000000, 67.000000, 62.000000, 30.000000, 49.000000, 86.000000, 37.000000, 84.000000, 18.000000, 38.000000, 18.000000, 44.000000, 21.000000, 86.000000, 99.000000, 13.000000, 79.000000, 84.000000, 9.000000, 48.000000, 74.000000, 52.000000, 99.000000, 80.000000,
    53.000000]
    ...
    DumpTensor: desc=5, addr=0, data_type=float16, position=UB, dump_size=32
    [35.000000, 41.000000, 41.000000, 22.000000, 84.000000, 49.000000, 60.000000, 0.000000, 90.000000, 14.000000, 67.000000, 80.000000, 16.000000, 46.000000, 16.000000, 83.000000, 6.000000, 70.000000, 97.000000, 28.000000, 97.000000, 62.000000, 80.000000, 22.000000, 53.000000, 37.000000, 23.000000, 58.000000, 65.000000, 28.000000, 4.000000,
    29.000000]
    
  • The following is an example of the printf printing. For details about the usage and restrictions of the printf API, see printf.
    1
    printf("fmt string %d", 0x123);
    

DumpTensor and printf affect the actual runtime performance of the operator. Therefore, this function is usually used in the debugging phase. You can disable the printing function as required. For details, see DumpTensor and printf.

Using msSanitizer to Detect Exceptions

msSanitizer is an exception detection tool based on Ascend AI Processor. It provides memory check, contention check, uninitialization check, and synchronization check in single-operator development scenarios.

For details, see msSanitizer (Anomaly Detection).

This function is supported only in the following scenarios:

Using msDebug for Operator Debugging

msDebug is an operator debugging tool for Ascend devices. It is used to debug operator programs running on NPUs and provides debugging methods for operator developers. msDebug can debug all Ascend operators, including Ascend C operators (Vector, Cube, and fused operators). The specific functions include breakpoint setting, variable and memory printing, single-step debugging, running interruption, core switching, program status check, debugging information display, and core dump file parsing. You can select the functions as needed. For details, see msDebug (Operator Debugging).