Performance Tuning

Performance Collection and Analysis Tool

After an executable program is generated using the BiSheng compiler, you can use the msProf tool to run the executable file generated in NPU mode to collect the performance data of the Ascend C operator executed on the AI processor for refined performance tuning.

Profiling performance data collection: Use the msProf tool to collect the performance data of the Ascend C operator executed on the AI processor.
Roofline bottleneck analysis: The visualize_data.bin file generated by msprof op can be visualized using MindStudio Insight. A Roofline bottleneck analysis chart can be used to build a processor performance model, which can be used to quickly evaluate the theoretical performance limit of an operator, allowing you to quickly identify bottlenecks.
Instruction pipeline chart analysis: The visualize_data.bin or trace.json file generated by msprof op simulator can be used for visualized display. An instruction pipeline chart displays timing relationship by instruction and associates with the call stack to quickly locate bottlenecks.

For details about how to use the performance tuning tool, see Operator Tuning (msProf).

Board Performance Tuning on the NPU

After an operator program is compiled into an executable program using the BiSheng compiler, you can use msprof op to collect performance data on the NPU. Take the SIMD programming scenario as an example. The procedure of using msProf to collect board performance data is as follows:

Compile the add operator sample to generate an executable file by referring to AI Core SIMD Programming.
The number following dav- indicates the NPU architecture version. Replace it with the actual one. For details about the architecture version number of each product model, see Table 1.
1
bisheng add_custom.asc -o add_custom --npu-arch=dav-2201
Use msprof op to call the operator executable file for performance data collection.
1
msprof op ./add_custom

View the performance data to learn about the performance bottlenecks of the current operator.

The following is an example of the profile data folder structure:
├──dump                       # Raw performance data, which you can ignore.
├──ArithmeticUtilization.csv  # Cube/Vector instruction cycle ratio. You are advised to optimize operator logic and reduce redundant computation instructions.
├──L2Cache.csv                # L2 cache hit rate, which affects MTE2. You are advised to properly plan the data transfer logic to increase the hit rate.
├──Memory.csv                 # UB, L1, and main memory read/write bandwidth (GB/s).
├──MemoryL0.csv               # L0A, L0B, and L0C read/write bandwidth (GB/s).
├──MemoryUB.csv               # Vector and scalar read/write bandwidth to/from the UB (GB/s).
├──OpBasicInfo.csv            # Basic operator information.
├──PipeUtilization.csv        # Pipe class instruction execution time and ratio. It is advised to optimize data movement logic to improve bandwidth utilization.
├──ResourceConflictRatio.csv  # Ratio of bank groups, bank conflicts, and resource conflicts in the UB to all instructions. You are advised to reduce or avoid read/write conflicts on the same bank or read/read conflicts on a bank group.
└──visualize_data.bin         # File for presentation in MindStudio Insight.

In the SIMT programming scenario, you only need to follow the AI Core SIMT compilation guide to compile the operator. After the executable file is generated, use the msProf tool to execute the program by following steps 2 and 3 above to obtain the operator execution performance data.

NPU Performance Simulation

On non-Ascend devices, after using the BiSheng compiler for simulation compilation to generate an executable program, you can use msprof op simulator to complete performance pipeline simulation. Currently, only the SIMD programming scenario is supported. The SIMT programming scenario is not supported.

Simulating and compiling operators using CMake

Flexibly control whether to enable simulation compilation for different targets. Modify CMakeLists and use target_link_libraries and target_link_directories to manually configure the link library and path.

find_package(ASC REQUIRED)
project(kernel_samples LANGUAGES ASC CXX)

add_executable(demo
    add_custom.asc
)

set_target_properties(demo PROPERTIES 
    LINK_FLAGS "-Wl,--disable-new-dtags"  ## Enable RPATH to pass the dependency directory because the simulation library depends on other non-linked .so files of the current directory.
)

target_link_libraries(demo PRIVATE
    runtime_camodel npu_drv_camodel  ## .so files to be linked for simulation compilation
    m
)

# Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
target_link_directories(demo PRIVATE
    ${INSTALL_DIR}/tools/simulator/dav_2201/lib  ## Directory where the simulation library is located. The dav_2201 directory name is related to the chip version.
)

target_compile_options(demo PRIVATE
    $<$<COMPILE_LANGUAGE:ASC>: --npu-arch=dav-2201>
)

The relationship between the directory where the simulation library is located and the NPU architecture version is as follows. The directory name uses an underscore to connect dav and the architecture version number.

**Table 1** Relationship between the simulator directory name and npu-arch
npu-arch	Simulator Directory Name
dav-2002	dav_2002
dav-2201	dav_2201
dav-3510	dav_3510

The following table describes the libraries on which simulation compilation depends. When simulation compilation is enabled, you need to preferentially link the libraries to ensure that the symbols of the simulation libraries are preferentially used, preventing exceptions such as core dumps during running.

**Table 2** Libraries on which simulation compilation depends
Name	Description
libruntime_camodel.so	CAModel simulation runtime library, which provides runtime function support for the NPU operator simulation environment.
libnpu_drv_camodel.so	Simulation driver library, which stubs real driver APIs, simulates real NPU driver behavior, and provides API simulation for hardware delivery.

Pass the variables CMAKE_ASC_RUN_MODE and CMAKE_ASC_ARCHITECTURES to CMake to enable simulation compilation globally. The following is an example command.
sim indicates to enable simulation compilation. The string following dav- is the NPU architecture version. Replace the string with the actual architecture version number.
```
cmake -B build -DCMAKE_ASC_RUN_MODE=sim -DCMAKE_ASC_ARCHITECTURES=dav-2201
```
Passing variables to CMake using the command line takes effect globally and enables the simulation mode for all targets in CMakeLists.

Simulating and compiling executable operator programs using the command line

# Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
# Set the simulator directory name.
export SIMULATOR_FOLDER=dav_2201
# Compile the operator object: bisheng -c [Operator source file] -o [Output product name] --npu-arch=[NPU architecture version] --run-mode=sim
bisheng -c add_custom.asc -o add_custom.asc.o --npu-arch=dav-2201 --run-mode=sim   
# Convert the object of the add operator into an executable program.
bisheng -Wl,--disable-new-dtags -L${INSTALL_DIR}/tools/simulator/${SIMULATOR_FOLDER}/lib -Wl,-rpath,${INSTALL_DIR}/tools/simulator/${SIMULATOR_FOLDER}/lib -lruntime_camodel -lnpu_drv_camodel -lm -lstdc++ -lascendcl -lascendc_runtime -lprofapi -lunified_dlog -lmmpa -lascend_dump -lc_sec -lerror_manager -lnpu_drv add_custom.asc.o -o add_custom

During compilation, in addition to the simulation libraries libruntime_camodel.so and libnpu_drv.so, you also need to link the libascendc_runtime.a, libruntime.so, libprofapi.so, libunified_dlog.so, libmmpa.so, libascend_dump.so, libc_sec.so, liberror_manager.so, and libascendcl.so libraries (for details, see Table 2), as well as third-party libraries libstdc++.so and libm.so.

Performance pipeline simulation

Use msprof op simulator to obtain simulation data.

msprof op simulator ./add_custom

Simulation data description

├──dump                    # Raw performance data, which you can ignore.
└──simulator               # Basic operator information.
   ├──core0.cubecore0
   ├──...
   ├──core23.cubecore0
   ├──trace.json           # File for presentation in Edge/Chrome Trace Viewer/Perfetto.
   └──visualize_data.bin   # File for presentation in MindStudio Insight.

Parent topic: Debugging and Tuning