Gelu

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product AI Core	√
Atlas inference product Vector Core	x
Atlas training product	x

Function Usage

In neural networks, GELU is an important activation function, which is inspired by ReLU and Dropout. Specifically, random regularization is introduced during activation. Below is the formula.

$\text{[math]}$

$\text{[math]}$ is simplified to obtain $\text{[math]}$

Prototype

Allocate the temporary space through the API framework.

template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void Gelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const uint32_t dataSize)

Pass the temporary space through the sharedTmpBuffer input parameter.

template <typename T, bool highPrecision = false, bool highPerformance = false>
__aicore__ inline void Gelu(const LocalTensor<T>& dstLocal, const LocalTensor<T>& srcLocal, const LocalTensor<uint8_t>& sharedTmpBuffer, const uint32_t dataSize)

Parameters

**Table 1** Template parameters
Parameter	Description
T	Data type of the operand. For the Atlas 350 Accelerator Card, the supported data types are half and float. For the Atlas A3 training product/Atlas A3 inference product, the supported data types are half and float. For the Atlas A2 training product/Atlas A2 inference product, the supported data types are half and float. For the Atlas inference product AI Core, the supported data types are half and float.
highPrecision	Whether to enable the high-precision mode to improve the computation accuracy. The default value is false, indicating that the high-precision mode is disabled. Note: The high-precision mode takes effect only when it is enabled for the half type. The value of this parameter does not affect the API precision and performance of the float type.
highPerformance	Whether to enable the high-performance mode to improve the computation efficiency. The default value is false, indicating that the high-performance mode is disabled. Note: Enabling the high-performance mode may decrease the precision when compared to the default setting, where both high-precision and high-performance modes are disabled. Enabling both the high-precision and high-performance modes may result in performance deterioration when compared to enabling only the high-performance mode. For the Atlas 350 Accelerator Card, this parameter is reserved but does not take effect. The value can be true or false. The API precision and performance are the same.

**Table 2** API parameters
Parameter	Input/Output	Description
dstLocal	Output	Destination operand. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
srcLocal	Input	Source operand. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. The source operand must have the same data type as the destination operand.
sharedTmpBuffer	Input	Temporary buffer. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. This parameter is used to store intermediate variables during complex computation and is provided by developers. For details about how to obtain the temporary space size (BufferSize), see GetGeluMaxMinTmpSize.
dataSize	Input	Number of elements involved in the computation.

Returns

None

Constraints

The tensor space of the source operand and destination operand can be reused.
The address of sharedTmpBuffer cannot overlap that of the source or destination operand.
For details about the operand address alignment requirements, see General Address Alignment Restrictions.
The input shape must be in ND format.

Examples

For details about the complete operator sample, see gelu operator sample.

// dstLocal: tensor for storing the Gelu computation result
// srcLocal: tensor for storing the Gelu computation input
// sharedTmpBuffer: tensor for storing the temporary buffer during Gelu computation

// Allocate the temporary space through the API framework, part of which is used for computation, with the number of elements involved in the computation being 32.
AscendC::Gelu<srcType, false>(dstLocal, srcLocal, 32);

// Pass the temporary space through the sharedTmpBuffer input parameter, part of which is used for computation, with the number of elements involved in computation being 32.
AscendC::Gelu<srcType, false>(dstLocal, srcLocal, sharedTmpBuffer, 32);

Result example:

Input (srcLocal):
[-2.56   -2.395  -2.23   -2.066  -1.9    -1.735  -1.571  -1.406
 -1.241  -1.076  -0.9116 -0.7466 -0.582  -0.417  -0.2522 -0.0874
  0.0774  0.2423  0.407   0.572   0.737   0.902   1.066   1.231
  1.396   1.561   1.726   1.891   2.055   2.22    2.385   2.55  ]
Output (dstLocal):
[-0.01295471 -0.01953125 -0.02836609 -0.03991699 -0.05453491 -0.07196045 -0.09130859 -0.11254883
 -0.13342285 -0.15185547 -0.16516113 -0.17004395 -0.16320801 -0.14111328 -0.10101318 -0.04067993
  0.04107666  0.14428711  0.26782227  0.40942383  0.56689453  0.7363281   0.9135742   1.0966797
  1.2822266   1.4677734   1.6533203   1.8349609   2.0136719   2.1914062   2.3632812   2.5390625 ]

Parent topic: Gelu APIs