[object Object]

[object Object][object Object]undefined[object Object]

• Function updates (compared with [object Object]):

  • The scale correction factors of the query and key are added, corresponding to [object Object] ($\alpha_q$) and [object Object] ($\alpha_{kv}$), respectively.
  • The optional input parameters (such as [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object]) are added, and the [object Object] parameter is changed from mandatory to optional.
  • The name and position of the [object Object] parameter are adjusted to match the current [object Object].

• Interface function:

  In inference scenarios, this operator performs the preprocessing computation for Multi-Head Latent Attention. The main computation process consists of five paths.

  • After multiplying the input $x$ with $W^{DQ}$ for downsampling and RmsNorm, the first path multiplies the result with $W^{UQ}$ and $W^{UK}$, followed by two upsampling operations, and then multiplies the result with the query scale correction factor $\alpha_q$ to obtain $q^N$. The second path multiplies the result with $W^{QR}$ and applies rotary position encoding (ROPE) to obtain $q^R$.
  • The third path is to multiply the input $x$ by $W^{DKV}$, perform downsampling and RmsNorm, and then multiply the result by the key scale correction factor $\alpha_{kv}$ to obtain $k^C$ and pass it to the cache.
  • The fourth path multiplies the input $x$ with $W^{KR}$, applies rotary position encoding (ROPE), and then stores the result into another cache to obtain $k^R$.
  • The fifth path processes the output $q^N$ through DynamicQuant to generate quantization parameters.
  • The weight parameters [object Object], [object Object], and [object Object] are required to be provided in NZ format.

• Formula:

  RmsNorm formula

  $$\text{RmsNorm}(x) = \gamma \cdot \frac{x_i}{\text{RMS}(x)}$$ $$\text{RMS}(x) = \sqrt{\frac{1}{N} \sum_{i=1}^{N} x_i^2 + \epsilon}$$

  The computation formula of the query, including downsampling, RmsNorm, and two upsampling operations.

  $$c^Q = \alpha_q\cdot\mathrm{RmsNorm}(x \cdot W^{DQ})$$ $$q^C = c^Q \cdot W^{UQ}$$ $$q^N = q^C \cdot W^{UK}$$

  Performs rotary position encoding (ROPE) to the query.

  $$q^R = \mathrm{ROPE}(c^Q \cdot W^{QR})$$

  The computation formula of the key, including downsampling and RmsNorm. The computation result is stored in the cache.

  $$c^{KV} = \alpha_{kv}\cdot\mathrm{RmsNorm}(x \cdot W^{DKV})$$ $$k^C = \mathrm{Cache}(c^{KV})$$

  Performs rotary position encoding (ROPE) to the key and stores the result in the cache.

  $$k^R = \mathrm{Cache}(\mathrm{ROPE}(x \cdot W^{KR}))$$

  Dequant Scale Query Nope calculation formula:

  $$\mathrm{dequantScaleQNope} = {\mathrm{RowMax}(\mathrm{abs}(q^{N})) / 127}$$ $$q^{N} = {\mathrm{round}(q^{N} / \mathrm{dequantScaleQNope})}$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]undefined

• Returns

  [object Object]: status code. For details, see .[object Object] The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]undefined

[object Object]

• Parameters

  [object Object]undefined

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computation.

• Shape field description

  [object Object]undefined

• Shape restrictions:

  • When BS fusion is used for [object Object], that is , [object Object]
    • The shape of [object Object] and [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • In the full quantization scenario, the shape of [object Object] is [object Object]. In other scenarios, the shape is [object Object].
  • When BS fusion is not used for [object Object], that is , [object Object]
    • The shape of [object Object] and [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • The shape of [object Object] is [object Object].
    • In the full quantization scenario, the shape of [object Object] is [object Object]. In other scenarios, the shape is [object Object].
  • One or more of the B, S, T, and Skv values can be 0. That is, input parameters related to the shape and B, S, T, and Skv values can be empty tensors. Other input parameters do not support empty tensors.
    • If the values of B, S, and T are 0, [object Object] and [object Object] output empty tensors, and [object Object] and [object Object] are not updated.
    • If the value of Skv is 0, [object Object], [object Object], and [object Object] are calculated normally, and [object Object] and [object Object] are not updated. That is, an empty tensor is output.

• Special constraints

  • In per-tile quantization mode, both [object Object] and [object Object] must be set to 1.
  • In per-tile quantization mode, [object Object] supports only [object Object], [object Object], and [object Object].
  • When the value of ckvkrRepoMode is 1, krCache must be an empty tensor (that is, the product of the shape is [object Object]).

• The [object Object] API supports the following scenarios:

  [object Object]

• In different quantization scenarios, the dtype combinations of parameters must meet the following conditions:

  [object Object]

[object Object]

The following example is for reference only. For details, see .