[object Object][object Object][object Object]undefined[object Object]

• Description: Implements normalization (Norm), Rotary Position Embedding (RoPE), and feature concatenation (Concat) for [object Object], [object Object], and [object Object] in the (multi-modal) transformer attention mechanism.

  • Currently, Norm supports layer normalization (LayerNorm) and layer normalization with affine transformation parameters (AFFINE LayerNorm).
  • RoPE supports the Interleave and Half types.
  • Concat can be performed in the sequence dimension with support for specifying different concatenation orders.

• The computation formulas (using [object Object] (video) and [object Object] (text) as examples) are as follows:

  $$hiddenState_q = \text{LayerNorm}(query, normQueryWeight, normQueryBias, eps) \\ hiddenState_{eq} = \text{LayerNorm}(encoderQuery, normEncoderQueryWeight, normEncoderQueryBias, eps) \\ concatedHiddenState = \text{Concat}(hiddenState_q, hiddenState_{eq}) \\ transposedHiddenState = \text{Transpose}(concatedHiddenState, (0, 2, 1, 3)) \\ hiddenState = \text{RoPE}(concatedHiddenState, ropeSin, ropeCos)$$

• Note:

  1. The input and output layouts are as follows: The shape of the input [object Object] is [object Object], and the shape of the output [object Object] is [object Object]. [object Object] indicates [object Object], [object Object] indicates [object Object], [object Object] indicates [object Object], and [object Object] indicates [object Object].

  2. LayerNorm has three modes (specified by [object Object]): [object Object], [object Object], and [object Object]. When [object Object]:

     $$hiddenState_q = query$$

     When [object Object]:

     $$queryMean_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D}query_{b,s,n} \\ queryVar_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D}(query-queryMean_{b,s,n})^2 \\ queryRstd_{b,s,n}= \frac{1}{\sqrt{queryVar_{b,s,n}+\epsilon}} \\ hiddenState_q = (query-queryMean)*queryRstd$$

     When [object Object] (based on the preceding formulas):

     $$hiddenState_q = normQueryWeight*hiddenState_q + normQueryBias$$

  3. Concat is performed in the sequence dimension. The concatenation order is specified by [object Object]. When [object Object], $hiddenState_q$ is before $hiddenState_{eq}$. When [object Object], $hiddenState_q$ is after $hiddenState_{eq}$.

  4. RoPE has three modes (specified by [object Object]):[object Object], [object Object], and [object Object]. When [object Object], the output is directly generated without transformation. In other cases, see the following:

     [object Object]

  5. The shape of the input [object Object] for RoPE is [object Object], where

     $$seqRope ≤ min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)$$

  6. In the training scenario, [object Object], [object Object], [object Object], and [object Object] are output for subsequent backward propagation.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• The data types of [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be the same.
• The value of [object Object] must be an even number in the range of [1, 1024].
• The value of [object Object] must be in the range of [1, Min([object Object] + [object Object], [object Object] + [object Object])].

[object Object]

The following example is for reference only. For details, see .