[object Object]

[object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas. This API is the weightNZ specialization version of .

• Formulas:

  [object Object]

  • Definition

    • ⋅ indicates matrix multiplication.
    • ⊙ indicates element-wise multiplication.
    • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
    • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
    • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

  • Input

    • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
    • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
    • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
    • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
    • $groupList∈\mathbb{N}^{E}$: Group index list of cumsum.

  • Output

    • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
    • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

  • Computation process

    • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

      [object Object]

    • 2. Perform the following computation based on the input parameters determined by grouping:

      $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\,\text{Broadcast}} \odot w\_scale_{i\,\text{Broadcast}}$

      $C_{i,act}, gate_{i} = split(C_{i})$

      $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

    • 3. Quantize the output.

      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

  [object Object][object Object]
  • Definition
    • ⋅ indicates matrix multiplication.
    • ⊙ indicates element-wise multiplication.
    • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
    • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
    • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
    • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
  • Input
    • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
    • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
    • $weightAssistMatrix∈\mathbb{R}^{E \times N}$: Auxiliary matrix used for matrix multiplication. For details about how to generate the auxiliary matrix, see the following description.
    • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
    • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
    • $groupList∈\mathbb{N}^{E}$: Group index list of cumsum.
  • Output
    • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
    • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
  • Computation process
    • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
      • The grouping logic is the same as that of A8W8.
    • 2. The computation process of generating the auxiliary matrix (weightAssistMatrix) is as follows. (Note that the computation of weightAssistMatrix is performed offline and is not completed inside the operator.)

      • For per-channel quantization ($w\_scale$ is 2D):

        $weightAssistMatrix_{i} = 8 × w\_scale × Σ_{k=0}^{K-1} weight[:,k,:]$

      • For per-group quantization ($w\_scale$ is 3D):

        $weightAssistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × w\_scale[:, ⌊k/num\_per\_group⌋, :])$

        Note: $num\_per\_group = K // K\_group\_num$

    • 3. Perform the following computation based on the input parameters determined by grouping:

      • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

      • 3.2. Enable per-channel or per-group quantization during matrix multiplication. Per-channel:

        $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

        $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

        Per-group:

        $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

      • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

        $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAssistMatrix_{i}) \odot x\_scale_{i}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

    • 3. Quantize the output.

      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

  [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Return

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a deterministic implementation.

• A8W8 scenario (A: activation matrix (left matrix); W: weight matrix (right matrix); 8: INT8)

  • 1. The length of the last axis of [object Object] cannot be greater than or equal to 65536.
  • 2. The length of the N-axis cannot exceed 10240.

• A8W4 scenario (A: activation matrix (left matrix); W: weight matrix (right matrix); 4: INT4)

  • 1. The length of the last axis of [object Object] cannot be greater than or equal to 20000.
  • 2. The length of the N-axis cannot exceed 10240.

[object Object]

The following example is for reference only. For details, see .