Functions

Function of the RELU Activation Type

Parameters
activationType = ACTIVATION_RELU
Function graph
Expression
$\text{[math]}$

Function of the GELU Activation Type

Tanh approximate computing
- Parameters
  - activationType = ACTIVATION_GELU
  - geluMode = TANH_MODE
- Function graph
- Expression
  $\text{[math]}$
- Restrictions
  Not supported for Atlas 200I/500 A2 inference products .
Non-approximate computing
- Parameters
  - activationType = ACTIVATION_GELU
  - geluMode = NONE_MODE
- Function graph
- Expression
  $\text{[math]}$
  
  is the cumulative distribution function of the Gaussian distribution.
  
  $\text{[math]}$

Function of the FAST_GELU Activation Type

Parameters
activationType = ACTIVATION_FAST_GELU
Function graph
Expression
$\text{[math]}$

Function of the SWISH Activation Type

Parameters
- activationType = ACTIVATION_SWISH
- scale
Function graph
It is related to the input parameter scale. When scale is 1:
Expression
$\text{[math]}$

Function of the Log Activation Type

Parameters
activationType = ACTIVATION_LOG
Expression
$\text{[math]}$

Function of the SWIGLU_FORWARD Activation Type

Parameters
- activationType = ACTIVATION_SWIGLU_FORWARD
- dim
Expression
a and b equalize the input tensor along the last dimension.

$\text{[math]}$
Restrictions
dim can be used to perform equalization only along the last dimension. -1 must be passed.

For the Atlas inference products , only 32-bit aligned data is supported.

Not supported for Atlas 200I/500 A2 inference products .

Function of the SWIGLU_BACKWARD Activation Type

Parameters
- activationType = ACTIVATION_SWIGLU_BACKWARD
- dim
Expression
x1: input tensor 1.

x2: input tensor 2

a and b equalize x2 along the last dimension.

$\text{[math]}$

$\text{[math]}$
Restrictions
dim can be used to perform equalization only along the last dimension. -1 must be passed.

Supported only by Atlas A2 training products / Atlas A2 inference products and Atlas A3 inference products / Atlas A3 training products .

SIGMOID Activation Type Functions

Parameters
activationType = ACTIVATION_SIGMOID
Function graph
Expression
$\text{[math]}$

Function of the FASTER_GELU_FORWARD Activation Type

Parameters
activationType = ACTIVATION_FASTER_GELU_FORWARD
Expression
$\text{[math]}$
Restrictions
Not supported for Atlas 200I/500 A2 inference products .

Parent topic: ActivationOperation