[object Object]

[object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas.

  Compared with the API, this API has the following new features:

  • Ascend 950PR/Ascend 950DT:
    • The MXFP8, MXFP4, and Pertoken quantization scenarios are added.
    • The field types of the weight, weightScale, and weightAssistMatrix parameters are changed to tensorlist. Select a proper API based on the actual situation.

• Formula:

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

        [object Object]

      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object][object Object]
    • Definition
      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
    • Input
      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $weightAssistMatrix∈\mathbb{R}^{E \times N}$: auxiliary matrix used for matrix multiplication. For details about how to generate the auxiliary matrix, see the following description.
      • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.
    • Output
      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
    • Computation process
      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. The computation process of generating the auxiliary matrix (weightAssistMatrix) is as follows. (Note that the computation of weightAssistMatrix is generated offline and used as the input, instead of being completed inside the operator.)

        • For per-channel quantization ($w\_scale$ is 2D):

          $weightAssistMatrix_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$

        • For per-group quantization ($w\_scale$ is 3D):

          $weightAssistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$

          Note: $num\_per\_group = K // K\_group\_num$

      • 3. Perform the following computation based on the input parameters determined by grouping:

        • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

        • 3.2. Enable per-channel or per-group quantization during matrix multiplication. Per-channel:

          $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          Per-group:

          $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

          $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

          $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAssistMatrix_{i}) \odot x\_scale_{i}$

          $C_{i,act}, gate_{i} = split(C_{i})$

          $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object][object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Input

      • $X∈\mathbb{Z_4}^{M \times K}$: activation matrix (left matrix), where $M$ indicates the total number of tokens and $K$ indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $smoothScale∈\mathbb{R}^{E \times N/2} (per channel) or \mathbb{R}^{E} (per tensor)$: smooth scaling factor, where E is the number of experts, and N is the output dimension.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

        $S_{i} = S_{i} \odot smoothScale_{i\ Broadcast}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]

  • Ascend 950PR/Ascend 950DT:

    [object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times ceil(K / 64) \times N \times 2}$: channel-wise scaling factor of the group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
      • $x\_scale∈\mathbb{R}^{M \times ceil(K / 64) \times 2}$: token-wise scaling factor of the activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M \times ceil((N / 2) / 64) \times 2}$: quantization scaling factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot xScale_{i\ Broadcast} \odot wScale_{i\ Broadcast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$, where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $shared\_exp = \left\lfloor \log_2(max_i(|S_i|)) \right\rceil - emax$

        $QScale = 2 ^ {shared\_exp}$

        $Q_i = quantize\_to\_element\_format(S_i/Qscale), \space i\space from\space 1\space to\space blocksize$

        • $emax$: exponent bit of the maximum positive regular number corresponding to the data type.

          [object Object]undefined

        • $blocksize$: number of elements to be quantized each time. Only 32 is supported.

    [object Object][object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scaling factor of the group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scaling factor of the activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

      • 2. Perform the following computation based on the input parameters determined by grouping:

           $C_{i} = (X_{i}\cdot W_{i} )\odot xScale_{i} \odot wScale_{i}$

           $C_{i,act}, gate_{i} = split(C_{i})$

           $S_{i}=Swish(C_{i,act})\odot gate_{i}$, where $Swish(x)=\frac{x}{1+e^{-x}}$

           $xScale_{i}$ indicates the quantization factor corresponding to the token.

      • 3. Quantize the output.

           $Q\_scale_{i} = \frac{max(|S_{i}|)}{max(type)}$

           $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

    [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
    • In the A4W4 scenario, the weight supports the transposition of NZ inputs. In other scenarios, only non-transposition is supported. INT32 is used for adaptation in the A8W4 and A4W4 scenarios. Actually, one INT32 is interpreted as eight INT4 data. The A8W8 scenario does not support the ND data format.
    • The dequantMode parameter is supported. In the A8W4 and A4W4 scenarios, the value can be 0 or 1. In the A8W8 scenario, the value can only be 0.
    • The dequantDtype and quantMode parameters are not supported.
    • x and weight do not support empty tensors.
    • When the weight NZ input is transposed, only the single-tensor mode is supported.
    • The weight, weightScale, and weightAssistMatrix support both the single-tensor scenario (the length of the tensor list is 1) and the multi-tensor scenario (the length of the tensor list is greater than 1).
  • Ascend 950PR/Ascend 950DT:
    • The weight supports only the ND format and can be transposed.
    • The dequantMode parameter is supported. In the MX quantization scenario, the value can be 2. In the Pertoken scenario, the value can be 0.
    • The dequantDtype parameter is supported. In the MX quantization scenario, the value can be 0. In the Pertoken scenario, the value can be 0, 1, or 27.
    • The quantMode parameter is supported. In the MX quantization scenario, the value can be 2. In the Pertoken scenario, the value can be 0.
    • Only the same value of dequantMode and quantMode is supported.
    • x and xScale support empty tensors with M being 0.
    • weight and weightScale support empty tensors with N being 0.
    • Currently, weight and weightScale support only the scenario where the length of the tensor list is 1.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a deterministic implementation.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • In the A8W8/A8W4/A4W4 quantization scenario, the following restrictions must be met:

    • Data type requirements

      [object Object]

    • The shape constraints must meet the requirements listed in the following table.

      [object Object]

  • In the A8W8 scenario, the length of the N-axis cannot exceed 10240, and the length of the last axis of [object Object] cannot be greater than or equal to 65536.

  • In the A8W4 scenario, the length of the N-axis cannot exceed 10240, and the length of the last axis of [object Object] cannot be greater than or equal to 20000.

  • In A4W4 scenarios, the size of the N axis must not exceed 10240, and the size of the last axis of [object Object] must be less than 20000.

  • In the multi-tensor scenario, that is, when the length of the tensor list is greater than 1, the shapes of weight, weightScale, and weightAssistMatrix need to be flattened based on the E dimension. For example, {(E, K, N)} needs to be changed to {E (K, N)}.

• Ascend 950PR/Ascend 950DT:

  • The first dimension of groupList supports a maximum of 1024 groups.

  • In the MX quantization scenario, the following constraints must be met:

    • Data type requirements

      [object Object]

    • The shape constraints must meet the requirements listed in the following table.

      [object Object]

    • The weightScale transpose attribute must be the same as that of weight.

    • In the MX quantization scenario, N must be 128-pixel aligned.

    • In the MXFP4 scenario, K cannot be 2.

    • In the MXFP4 scenario, K must be an even number. If the output data type is FLOAT4_E2M1, N must be an even number greater than or equal to 4.

  • In the per-token quantization scenario, the following constraints must be met:

    • Data type requirements

      [object Object]

    • The shape constraints must meet the following requirements:

      [object Object]

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

• Ascend 950PR/Ascend 950DT:

  [object Object]