[object Object]

[object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas. This API is the weightNZ specialization version of . The difference between this API and aclnnGroupedMatmulSwigluQuantV2 is that the weight parameter is forcibly regarded as the FRACTAL_NZ format in this scenario.

• Formulas:

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Inputs

      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where $E$ indicates the number of routed experts, $K$ indicates the feature dimension, and $N$ indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Outputs

      • $Q∈\mathbb{Z_8}^{M \times N/2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Calculation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

        [object Object]

      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]
    • Definition
      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
    • Inputs
      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where $E$ indicates the number of routed experts, $K$ indicates the feature dimension, and $N$ indicates the output dimension.
      • $weightAssistMatrix∈\mathbb{R}^{E \times N}$: auxiliary matrix for matrix multiplication (the computation process for generating the auxiliary matrix is described below).
      • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.
    • Outputs
      • $Q∈\mathbb{Z_8}^{M \times N/2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
    • Calculation Process:
      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. The computation process of generating the auxiliary matrix (weightAssistMatrix) is as follows. (Note that the computation of weightAssistMatrix is generated offline and used as the input, instead of being completed inside the operator.)

        • For per-channel quantization ($w\_scale$ is 2D):

          $weightAssistMatrix_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$

        • For per-group quantization ($w\_scale$ is 3D):

          $weightAssistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$

          Note: $num\_per\_group = K // K\_group\_num$

      • 3. Perform the following computation based on the input parameters determined by grouping:

        • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

        • 3.2. Enable per-channel or per-group quantization during matrix multiplication. Per-channel:

          $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          Per-group:

          $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

          $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

          $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAssistMatrix_{i}) \odot x\_scale_{i}$

          $C_{i,act}, gate_{i} = split(C_{i})$

          $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Inputs

      • $X∈\mathbb{Z_4}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where $E$ indicates the number of routed experts, $K$ indicates the feature dimension, and $N$ indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $smoothScale∈\mathbb{R}^{E \times N/2} (per channel) or \mathbb{R}^{E} (per tensor)$: smooth scaling factor, where E is the number of experts, and N is the output dimension.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Outputs

      • $Q∈\mathbb{Z_8}^{M \times N/2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Calculation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

        $S_{i} = S_{i} \odot smoothScale_{i\ Broadcast}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object] and [object Object]Atlas A2 training series products/Atlas A2 inference series products[object Object]:
    • [object Object]The weight is forcibly regarded as the FRACTAL_NZ format.[object Object]
    • The weight supports transposition in the A4W4 scenario, and supports only non-transposition in other scenarios. INT32 is used for adaptation in the A8W4 and A4W4 scenarios. In practice, one INT32 is interpreted as eight INT4 data. The ND data format is not supported in the A8W8 scenario.
    • The dequantMode parameter is supported. In the A8W4 and A4W4 scenarios, the value can be 0 or 1. In the A8W8 scenario, the value can only be 0.
    • The dequantDtype and quantMode parameters are not supported.
    • x and weight do not support empty tensors.
    • When the weight is transposed in the NZ format, only the single-tensor mode is supported.
    • weight, weightScale, and weightAssistMatrix support both single-tensor scenarios (the length of the tensor list is 1) and multi-tensor scenarios (the length of the tensor list is greater than 1).

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
  • In the A8W8/A8W4/A4W4 quantization scenario, the following restrictions must be met:

    • Data type requirements

      [object Object]

    • The shape constraints must meet the requirements listed in the following table.

      [object Object]

    • In A8W8 scenarios, the size of the N axis must not exceed 10240, and the size of the last axis of [object Object] must be less than 65536.

    • In A8W4 scenarios, the size of the N axis must not exceed 10240, and the size of the last axis of [object Object] must be less than 20000.

    • In A4W4 scenarios, the size of the N axis must not exceed 10240, and the size of the last axis of [object Object] must be less than 20000.

    • In the multi-tensor scenario, that is, when the length of the tensor list is greater than 1, the shapes of weight, weightScale, and weightAssistMatrix need to be flattened according to the dimension of E. For example, {(E, K, N)} needs to be changed to {E (K, N)}.

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]