Statement: This document is licensed under the license. Any forwarding, quoting, modification, or other operations performed on this document shall comply with the license agreement.

[object Object][object Object][object Object]undefined[object Object]

• Description: Implements normalization (Norm), Rotary Position Embedding (RoPE), and feature concatenation (Concat) for [object Object], [object Object], and [object Object] in the (multi-modal) transformer attention mechanism.

  • Normalization (Norm) supports layer normalization (LayerNorm), affine layer normalization (AFFINE LayerNorm), root mean square normalization (RmsNorm), and affine root mean square normalization (AFFINE RmsNorm).
  • RoPE supports the Interleave and Half types.
  • Concat can be performed in the sequence dimension with support for specifying different concatenation orders.

• The computation formulas (using [object Object] (video) and [object Object] (text) as examples) are as follows:

  $$hiddenState_q = \text{Norm}(query, normQueryWeight, normQueryBias, eps) \\ hiddenState_{eq} = \text{Norm}(encoderQuery, normEncoderQueryWeight, normEncoderQueryBias, eps) \\ concatedHiddenState = \text{Concat}(hiddenState_q, hiddenState_{eq}) \\ transposedHiddenState = \text{Transpose}(concatedHiddenState, (0, 2, 1, 3)) \\ hiddenState = \text{RoPE}(concatedHiddenState, ropeSin, ropeCos)$$

• Note:

  1. The input and output layouts are as follows: The shape of the input [object Object] is [object Object], and the shape of the output [object Object] is [object Object]. [object Object] indicates [object Object], [object Object] indicates [object Object], [object Object] indicates [object Object], and [object Object] indicates [object Object].

  2. There are five types of normalization (normType): NONE(0), LAYER_NORM(1), LAYER_NORM_AFFINE(2), RMS_NORM(3), and RMS_NORM_AFFINE(4). When [object Object]:

     $$hiddenState_q = query$$

     When [object Object]:

     $$queryMean_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D}query_{b,s,n} \\ queryVar_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D}(query-queryMean_{b,s,n})^2 \\ queryRstd_{b,s,n}= \frac{1}{\sqrt{queryVar_{b,s,n}+\epsilon}} \\ hiddenState_q = (query-queryMean)*queryRstd$$

     When normType is set to LAYER_NORM_AFFINE, the following is added:

     $$hiddenState_q = normQueryWeight*hiddenState_q + normQueryBias$$

     When normType is set to RMS_NORM:

     $$queryMs = \frac{1}{D}\sum_{i=0}^{D}(query_{b,s,n})^2 \\ queryRms = \frac{1}{\sqrt{queryMs+\epsilon}} \\ hiddenState_q = query * queryRms$$

     When normType is set to RMS_NORM_AFFINE, the following is added:

     $$hiddenState_q = normQueryWeight*hiddenState_q$$

  3. Concat is performed in the sequence dimension. The concatenation order is specified by [object Object]. When [object Object], $hiddenState_q$ is before $hiddenState_{eq}$. When [object Object], $hiddenState_q$ is after $hiddenState_{eq}$.

  4. RoPE has three modes (specified by [object Object]):[object Object], [object Object], and [object Object]. When [object Object], the output is directly generated without transformation. In other cases, see the following:

     [object Object]

  5. The shape of the input [object Object] for RoPE is [object Object], where

     $$seqRope <= min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)$$

  6. In the training scenario, [object Object], [object Object], [object Object], and [object Object] are output for subsequent backward propagation.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• The data types of [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be the same.
• The value of [object Object] must be an even number in the range of [1, 1024].
• The value of [object Object] must be in the range of [1, Min([object Object] + [object Object], [object Object] + [object Object])].

[object Object]

The following example is for reference only. For details, see .