FasterGelu

Applicability

Function

In neural networks, GELU is an important activation function, which is inspired by ReLU and Dropout. Specifically, random regular expression is introduced during activation. To reduce the compute demand of GELU, versions such as FastGelu are proposed in the industry. This API is a simplified version of FastGelu, and the simplified formula greatly improves the computing performance. Below is the formula.

is simplified to obtain

Prototype

• Pass to the temporary space through the sharedTmpBuffer input parameter.

  1 2	template <typename T, bool highPrecision = false, bool highPerformance = false> __aicore__ inline void FasterGelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const LocalTensor<uint8_t>& sharedTmpBuffer, const uint32_t dataSize)

• Allocate the temporary space through the API framework.

  1 2	template <typename T, bool highPrecision = false, bool highPerformance = false> __aicore__ inline void FasterGelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const uint32_t dataSize)

Due to the complex mathematical computation involved in the internal implementation of this API, extra temporary space is required to store intermediate variables generated during computation. The temporary space can be passed by developers through the sharedTmpBuffer input parameter or allocated through the API framework.

• When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated and deallocated, improving the flexibility and buffer utilization.
• When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the space.

If sharedTmpBuffer is used, developers must allocate space for the tensor. If the API framework is used, developers must reserve the temporary space. To obtain the temporary space (BufferSize) to be reserved, use the API provided in GetGeluMaxMinTmpSize.

Parameters

Returns

None

Restrictions

• The tensor space of the source operand and destination operand can be reused.
• For details about the operand address alignment requirements, see General Address Alignment Restrictions.
• Currently, only the ND format is supported.
• The address of sharedTmpBuffer must not overlap that of the source or destination operand.

Example

For a complete example, see fastergelu.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

#include "kernel_operator.h"

template <typename srcType>
class KernelFasterGelu
{
public:
    __aicore__ inline KernelFasterGelu() {}
    __aicore__ inline void Init(GM_ADDR src_gm, GM_ADDR dst_gm, uint32_t inputSize)
    {
        dataSize = inputSize;
        src_global.SetGlobalBuffer(reinterpret_cast<__gm__ srcType *>(src_gm), dataSize);
        dst_global.SetGlobalBuffer(reinterpret_cast<__gm__ srcType *>(dst_gm), dataSize);
        pipe.InitBuffer(inQueueX, 1, dataSize * sizeof(srcType));
        pipe.InitBuffer(outQueue, 1, dataSize * sizeof(srcType));
    }
    __aicore__ inline void Process()
    {
        CopyIn();
        Compute();
        CopyOut();
    }

private:
    __aicore__ inline void CopyIn()
    {
        AscendC::LocalTensor<srcType> srcLocal = inQueueX.AllocTensor<srcType>();
        AscendC::DataCopy(srcLocal, src_global, dataSize);
        inQueueX.EnQue(srcLocal);
    }
    __aicore__ inline void Compute()
    {
        AscendC::LocalTensor<srcType> dstLocal = outQueue.AllocTensor<srcType>();
        AscendC::LocalTensor<srcType> srcLocal = inQueueX.DeQue<srcType>();
        AscendC::FasterGelu(dstLocal, srcLocal, dataSize);
        // AscendC::FasterGelu<srcType, true, false>(dstLocal, srcLocal, dataSize); Enable the high-precision mode.
        // AscendC::FasterGelu<srcType, false, true>(dstLocal, srcLocal, dataSize); Enable the high-performance mode.
        outQueue.EnQue<srcType>(dstLocal);
        inQueueX.FreeTensor(srcLocal);
    }
    __aicore__ inline void CopyOut()
    {
        AscendC::LocalTensor<srcType> dstLocal = outQueue.DeQue<srcType>();
        AscendC::DataCopy(dst_global, dstLocal, dataSize);
        outQueue.FreeTensor(dstLocal);
    }

private:
    AscendC::GlobalTensor<srcType> src_global;
    AscendC::GlobalTensor<srcType> dst_global;

    AscendC::TPipe pipe;
    AscendC::TQue<AscendC::TPosition::VECIN, 1> inQueueX;
    AscendC::TQue<AscendC::TPosition::VECOUT, 1> outQueue;

    uint32_t dataSize = 0;
};

template <typename dataType>
__aicore__ void kernel_FasterGelu_operator(GM_ADDR src_gm, GM_ADDR dst_gm, uint32_t dataSize)
{
    KernelFasterGelu<dataType> op;
    op.Init(src_gm, dst_gm, dataSize);
    op.Process();
}

1
2

Input data (srcLocal): [-1.83887 -3.60742 3.12891 -0.620605 2.0625 -2.77344 -0.04422 -3.54297 -3.16211 2.67383 1.3291 -1.57617 -0.0123901 3.77539 -1.61621 -0.616699]
Output data (dstLocal): [-0.0769653 -0.00775528 3.11328 -0.160034 2.00195 -0.0244446 -0.021286 -0.00849152 -0.0144653 2.64453 1.20312 -0.100769 -0.00613022 3.76758 -0.0969238 -0.159912]