• To compile and execute operator APIs, ensure that the basic environment has been set up, including the driver, firmware, CANN software package, and ops package.

• For details about the operator API call process and compilation and running operations, see "Single-Operator Calling > Single-Operator API Execution > Sample Code for Calling aclnn APIs" in .

Assume that the development environment and operating environment are deployed on the same server (equipped with the AI processor). In this scenario, code development and code running are performed on the same machine. The following uses the FlashAttentionScore operator as an example. The calling logic, process, and compilation script of other operators are similar to those of the FlashAttentionScore operator. You should modify the API calling script (*.cpp) and compilation script (CMakeLists) as required.

• Sample code

  The FlashAttentionScore operator is used to implement self-attention calculation.

  • If psetype is set to 1, addition is required before multiplication.
  • If psetype is not set to 1, multiplication is required before addition.

• Formula:

  The forward propagation formula for attention is as follows:

  • If psetype is set to 1, the formula is as follows:

    $$attention\_out = Dropout(Softmax(Mask(scale*(pse+query*key^T), atten\_mask)), keep\_prob)*value$$

  • If psetype is not set to 1, the formula is as follows:

    $$attention\_out=Dropout(Softmax(Mask(scale*(query*key^T) + pse),atten\_mask),keep\_prob)*value$$

  You can obtain the sample code from the document and name the code file test_flash_attention_score.cpp.

• CMakeLists file

  The following is an example of the CMake file, which should be modified as required.

  [object Object]

  When compiling and generating operator samples in this project, you need to link the libopapi_math.so dynamic library. This is because some L0 APIs are called during operator implementation, and these APIs are encapsulated in libopapi_math.so. Therefore, you need to explicitly declare this dependency during compilation and linking.

  Operators with integrated and parallel collective communication and MatMul computation are collectively referred to as merged compute and communication operators (abbreviated as MC2 operators), including AllGatherMatmul, AlltoAllAllGatherBatchMatMul, BatchMatMulReduceScatterAlltoAll, MatmulAllReduce, MatmulAllReduceAddRmsNorm, and MatmulReduceScatter. When these operator APIs are called, multi-threading and Huawei Collective Communication Library (HCCL) are generally involved. Therefore, the following content needs to be imported to the CMake file to ensure successful compilation:

  [object Object]

  You can use the find_package(Threads REQUIRED) of CMake to search for the thread library. The command can automatically link the header files on which the thread library depends or the library files on which the thread library indirectly depends.

1. Prepare the calling code (*.cpp) and compilation script (CMakeLists.txt) of the operator.

2. Set environment variables.

   After installing the CANN software, log in to the environment as the CANN running user and run the following command to make the environment variables take effect:

   [object Object]

   ${INSTALL_DIR} indicates the CANN component directory, which should be replaced with the actual directory.

3. Compile and run the script.

   • Go to the directory where CMakeLists.txt is stored and run the following command to create the build folder to store the generated compilation file.

     [object Object]

   • Go to the build directory, run the cmake command to compile the code, and then run the make command to generate an executable file.

     [object Object]

     After the compilation is successful, the opapi_test executable file is generated in the bin folder in the build directory.

   • Go to the bin folder and run the executable file opapi_test.

     [object Object]

     The following uses the running result of the FlashAttentionScore operator as an example:

     [object Object]