Documentation

FasterGeluV2

Function Usage

In neural networks, GELU is an important activation function, which is inspired by ReLU and Dropout. Specifically, random regular expression is introduced during activation. To reduce the computing power requirements of GELU, versions such as FastGeluV2 are proposed in the industry. The specific calculation formula is as follows, where PAR represents the number of elements that can be processed by Vector Unit in one iteration.

where .

Prototype

  • Pass the temporary space through the sharedTmpBuffer input parameter.
    1
2
    template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void FasterGeluV2(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const LocalTensor<uint8_t>& sharedTmpBuffer, const uint32_t dataSize)
  • Allocate the temporary space through the API framework.
    1
2
    template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void FasterGeluV2(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const uint32_t dataSize)

Due to the complex mathematical computation involved in the internal implementation of this API, additional temporary space is required to store intermediate variables generated during computation. The temporary space can be passed by developers through the sharedTmpBuffer input parameter or allocated through the API framework.

  • When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated and deallocated, improving the flexibility and buffer utilization.
  • When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the space.

If sharedTmpBuffer is passed, developers must allocate space for the tensor. If the API framework is used, developers must reserve the temporary space. To obtain the size of the temporary space (BufferSize) to be reserved, use the API provided in FasterGeluV2 Tiling.

Parameters

Table 1 Parameters in the template

Parameter

Description

T

Data type of the operand.

highPrecision

Whether to enable the high-precision API to improve the computing accuracy. The default value is false, indicating that the function is disabled.

highPerformance

Whether to enable the high-performance API to improve the computing efficiency. The default value is false, indicating that the function is disabled. Note: Enabling the high-performance mode can result in a decrease in precision when compared to disabling the high-precision and high-performance modes by default. Enabling both the high-precision and high-performance modes may result in a decrease in performance when compared to enabling only the high-performance mode.
Table 2 API parameters

Parameter

Input/Output

Description

dstLocal

Output

Destination operand.

The type is LocalTensor, and the supported TPosition is VECIN, VECCALC, or VECOUT.

srcLocal

Input

Source operand.

The type is LocalTensor, and the supported TPosition is VECIN, VECCALC, or VECOUT.

The source operand must have the same data type as the destination operand.

sharedTmpBuffer

Input

Temporary space.

The type is LocalTensor, and the supported TPosition is VECIN, VECCALC, or VECOUT.

The data type of this operand is fixed at uint8_t.

This parameter is used to store intermediate variables during complex computation and is provided by developers.

For details about how to obtain the temporary space size (BufferSize), see GetGeluMaxMinTmpSize.

dataSize

Input

Number of actually computed data elements. Value range: dataSize ∈ [0, min(srcLocal.GetSize(), dstLocal.GetSize())].

Returns

None

Availability

Constraints

  • The tensor space of the source operand and destination operand can be reused.
  • For details about the alignment requirements of the operand address offset, see General Restrictions.
  • Currently, only the ND format is supported.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#include "kernel_operator.h"

template <typename srcType>
class KernelFasterGelu
{
public:
    __aicore__ inline KernelFasterGelu() {}
    __aicore__ inline void Init(GM_ADDR srcGm, GM_ADDR dstGm, uint32_t inputSize)
    {
        dataSize = inputSize;
        src_global.SetGlobalBuffer(reinterpret_cast<__gm__ srcType *>(srcGm), dataSize);
        dst_global.SetGlobalBuffer(reinterpret_cast<__gm__ srcType *>(dstGm), dataSize);
        pipe.InitBuffer(inQueueX, 1, dataSize * sizeof(srcType));
        pipe.InitBuffer(outQueue, 1, dataSize * sizeof(srcType));
    }
    __aicore__ inline void Process()
    {
        CopyIn();
        Compute();
        CopyOut();
    }

private:
    __aicore__ inline void CopyIn()
    {
        AscendC::LocalTensor<srcType> srcLocal = inQueueX.AllocTensor<srcType>();
        AscendC::DataCopy(srcLocal, src_global, dataSize);
        inQueueX.EnQue(srcLocal);
    }
    __aicore__ inline void Compute()
    {
        AscendC::LocalTensor<srcType> dstLocal = outQueue.AllocTensor<srcType>();
        AscendC::LocalTensor<srcType> srcLocal = inQueueX.DeQue<srcType>();
        AscendC::FasterGeluV2(dstLocal, srcLocal, dataSize);
        // AscendC::FasterGeluV2<srcType, true, false>(dstLocal, srcLocal, dataSize); Enable the high-precision mode.
        // AscendC::FasterGeluV2<srcType, false, true>(dstLocal, srcLocal, dataSize); Enable the high-performance mode.
        outQueue.EnQue<srcType>(dstLocal);
        inQueueX.FreeTensor(srcLocal);
    }
    __aicore__ inline void CopyOut()
    {
        AscendC::LocalTensor<srcType> dstLocal = outQueue.DeQue<srcType>();
        AscendC::DataCopy(dst_global, dstLocal, dataSize);
        outQueue.FreeTensor(dstLocal);
    }

private:
    AscendC::GlobalTensor<srcType> src_global;
    AscendC::GlobalTensor<srcType> dst_global;
    AscendC::TPipe pipe;
    AscendC::TQue<AscendC::QuePosition::VECIN, 1> inQueueX;
    AscendC::TQue<AscendC::QuePosition::VECOUT, 1> outQueue;
    uint32_t dataSize = 0;
};

template <typename dataType>
__aicore__ void kernel_FasterGelu_operator(GM_ADDR srcGm, GM_ADDR dstGm, uint32_t dataSize)
{
    KernelFasterGelu<dataType> op;
    op.Init(srcGm, dstGm, dataSize);
    op.Process();
}

Result example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Input data (srcLocal):
[-1.251   1.074  -6.137  -9.67   -5.066  -9.44   -3.588  -5.758  -7.484
 -5.35   -9.62   -4.33   -6.66   -3.732   0.0841 -8.59   -6.3    -4.62
 -3.059  -8.34   -8.24   -7.617  -7.93   -3.592  -3.268  -5.406  -9.49
  5.633  -5.3    -9.36   -6.715  -5.727 ]
Output data (dstLocal):
[-0.1411  0.916  -0.     -0.     -0.     -0.     -0.     -0.     -0.
 -0.     -0.     -0.     -0.     -0.      0.0486 -0.     -0.     -0.
 -0.     -0.     -0.     -0.     -0.     -0.     -0.     -0.     -0.
  5.633  -0.     -0.     -0.     -0.    ]
Parent topic: Gelu