Parameters

activationType

ActivationType

ACTIVATION_UNDEFINED

Activation type.

• ACTIVATION_UNDEFINED: undefined. This default type is unavailable. You need to set this parameter.
• ACTIVATION_RELU: ReLU activation type.
• ACTIVATION_GELU: GELU activation type.
• ACTIVATION_FAST_GELU: GELU activation type for fast operation.
• ACTIVATION_SWISH: SWISH activation type.
• ACTIVATION_LOG: LOG activation type.
• ACTIVATION_SWIGLU_FORWARD: SwiGLU forward activation function.
• ACTIVATION_SWIGLU_BACKWARD: Swiglu_forward backward activation function, which is used for gradient calculation.
• ACTIVATION_SIGMOID: Sigmoid activation type.
• ACTIVATION_FASTER_GELU_FORWARD: FASTER_GELU_FORWARD activation type.
• ACTIVATION_MAX: maximum enumerated value, not an activation type.

scale

float

1.0f

Scaling coefficient in the Swish activation function

dim

int32_t

-1

For the SwiGLU forward and backward activation functions, split along the last dimension. That is, only -1 can be passed.

geluMode

GeLUMode

TANH_MODE

GELU model selection.

• TANH_MODE: Tanh approximate computing.
• NONE_MODE: non-approximate computing.

rsv[8]

uint8_t

{0}

Reserved