Functions

ReLU Activation

Parameters
activationType = ACTIVATION_RELU
Function graph
Expression
$\text{[math]}$

GELU Activation

Tanh approximate computing
- Parameters
  - activationType = ACTIVATION_GELU
  - geluMode = TANH_MODE
- Function graph
- Expression
  $\text{[math]}$
- Constraints
  Not supported for Atlas 200I/500 A2 inference products .
Non-approximate computing
- Parameters
  - activationType = ACTIVATION_GELU
  - geluMode = NONE_MODE
- Function graph
- Expression
  $\text{[math]}$
  
  is the cumulative distribution function of the Gaussian distribution.
  
  $\text{[math]}$

FAST_GELU Activation

Parameters
activationType = ACTIVATION_FAST_GELU
Function graph
Expression
$\text{[math]}$

SWISH Activation

Parameters
- activationType = ACTIVATION_SWISH
- scale
Function graph
Related to the input parameter scale. When scale is 1:
Expression
$\text{[math]}$

LOG Activation

Parameters
activationType = ACTIVATION_LOG
Expression
$\text{[math]}$

SWIGLU_FORWARD Activation

Parameters
- activationType = ACTIVATION_SWIGLU_FORWARD
- dim
Expression
a and b equalize the input tensor along the last dimension.

$\text{[math]}$
Constraints
dim can be used to perform equalization only along the last dimension. -1 must be passed.

For Atlas inference products , only 32-bit aligned data is supported.

Not supported for Atlas 200I/500 A2 inference products .

SWIGLU_BACKWARD Activation

Parameters
- activationType = ACTIVATION_SWIGLU_BACKWARD
- dim
Expression
x1: input tensor 1.

x2: input tensor 2

a and b equalize x2 along the last dimension.

$\text{[math]}$

$\text{[math]}$
Constraints
dim can be used to perform equalization only along the last dimension. -1 must be passed.

Supported only by Atlas A2 training products / Atlas A2 inference products and Atlas A3 inference products / Atlas A3 training products .

SIGMOID Activation

Parameters
activationType = ACTIVATION_SIGMOID
Function graph
Expression
$\text{[math]}$

FASTER_GELU_FORWARD Activation

Parameters
activationType = ACTIVATION_FASTER_GELU_FORWARD
Expression
$\text{[math]}$
Constraints
Not supported for Atlas 200I/500 A2 inference products .

Parent topic: ActivationOperation