FasterGelu

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product / Atlas A3 inference product	√
Atlas A2 training product / Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product AI Core	√
Atlas inference product Vector Core	x
Atlas training product	x

Function Usage

In neural networks, GELU is an important activation function, which is inspired by ReLU and Dropout. Specifically, random regular expression is introduced during activation. To reduce the compute demand of GELU, versions such as FastGelu are proposed in the industry. This API FasterGelu is a simplified version of FastGelu, and the simplified formula greatly improves the computing performance. Below is the formula.

$\text{[math]}$

$\text{[math]}$ is simplified to obtain

Prototype

Pass the temporary space through the sharedTmpBuffer input parameter.

        
             template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void FasterGelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const LocalTensor<uint8_t>& sharedTmpBuffer, const uint32_t dataSize)

Allocate the temporary space through the API framework.

        
             template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void FasterGelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const uint32_t dataSize)

Due to the complex mathematical computation involved in the internal implementation of this API, extra temporary space is required to store intermediate variables generated during computation. The temporary space can be passed by developers through the sharedTmpBuffer input parameter or allocated through the API framework.

When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization.
When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the temporary space.

If sharedTmpBuffer is used, developers must allocate space for the tensor. If the API framework is used, developers must reserve the temporary space. To obtain the size of the temporary space (BufferSize) to be reserved, use the API provided in GetGeluMaxMinTmpSize.

Parameters

**Table 1** Template parameters
Parameter	Description
T	Data type of the operand. For the Atlas 350 Accelerator Card, the supported data types are half and float. For the Atlas A3 training product / Atlas A3 inference product , the supported data types are half and float. For the Atlas A2 training product / Atlas A2 inference product , the supported data types are half and float. For the Atlas inference product AI Core, the supported data types are half and float.
highPrecision	Whether to enable the high-precision mode to improve the computation accuracy. The default value is false, indicating that the high-precision mode is disabled. Note: The high-precision mode takes effect only when it is enabled for the half type. The value of this parameter does not affect the API precision and performance of the float type.
highPerformance	Whether to enable the high-performance mode to improve the computation efficiency. The default value is false, indicating that the high-performance mode is disabled. Note: Enabling the high-performance mode may decrease the precision when compared to the default setting, where both high-precision and high-performance modes are disabled. Enabling both the high-precision and high-performance modes may result in performance deterioration when compared to enabling only the high-performance mode. For the Atlas 350 Accelerator Card, this parameter is reserved but does not take effect. The value can be true or false. The API precision and performance are the same.

**Table 2** API parameters
Parameter	Input/Output	Description
dstLocal	Output	Destination operand. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
srcLocal	Input	Source operand. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. The source operand must have the same data type as the destination operand.
sharedTmpBuffer	Input	Temporary buffer. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. This parameter is used to store intermediate variables during complex computation and is provided by developers. For details about how to obtain the temporary space size (BufferSize), see GetGeluMaxMinTmpSize.
dataSize	Input	Number of elements involved in the computation.

Returns

None

Constraints

The tensor space of the source operand and destination operand can be reused.
For details about the operand address alignment requirements, see General Address Alignment Restrictions.
Currently, only the ND format is supported.
The address of sharedTmpBuffer cannot overlap that of the source or destination operand.

Examples

For a complete call sample, see FasterGelu operator sample.

      
           // dstLocal: output tensor
// srcLocal: input tensor
// dataSize: number of elements involved in the computation

// Disable the high-precision and high-performance modes by default.
AscendC::FasterGelu(dstLocal, srcLocal, dataSize);
// Enable the high-precision mode. The high-precision mode takes effect only when srcType is half.
AscendC::FasterGelu<srcType, true, false>(dstLocal, srcLocal, dataSize);
// Enable the high-performance mode.
AscendC::FasterGelu<srcType, false, true>(dstLocal, srcLocal, dataSize);

Result example:

       
            Input (srcLocal): [-1.83887 -3.60742 3.12891 -0.620605 2.0625 -2.77344 -0.04422 -3.54297 -3.16211 2.67383 1.3291 -1.57617 -0.0123901 3.77539 -1.61621 -0.616699]
Output (dstLocal): [-0.0769653 -0.00775528 3.11328 -0.160034 2.00195 -0.0244446 -0.021286 -0.00849152 -0.0144653 2.64453 1.20312 -0.100769 -0.00613022 3.76758 -0.0969238 -0.159912]

Parent topic: Gelu APIs