[object Object]

[object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas. Compared with , this API changes the field type of the [object Object], [object Object], and [object Object] parameters to tensor list. Select an appropriate API as required.
• Formula:
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:[object Object]

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

        [object Object]

      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ BroadCast} \odot w\_scale_{i\ BroadCast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]
    • Definition
      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
    • Input
      • $X∈\mathbb{Z_8}^{M \times K}$: activation matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $weightAsistMatrix∈\mathbb{R}^{E \times N}$: auxiliary matrix for matrix multiplication (the computation process for generating the auxiliary matrix is described below).
      • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the activation matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of cumsum or count.
    • Output
      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
    • Computation process
      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. Compute the auxiliary matrix ([object Object]). (Note that the computation is performed offline and provided as an input, rather than being executed within the operator.)

        • For per-channel quantization ($w\_scale$ is 2D):

          $weightAsistMatrix_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$

        • For per-group quantization ($w\_scale$ is 3D):

          $weightAsistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$

          Note: $num\_per\_group = K // K\_group\_num$

      • 3. Perform the following computation based on the input parameters determined by grouping:

        • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

        • 3.2. Enable per-channel or per-group quantization during matrix multiplication.

          Per-channel:

          $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          Per-group:

          $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

          $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

          $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAsistMatrix_{i}) \odot x\_scale_{i}$

          $C_{i,act}, gate_{i} = split(C_{i})$

          $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

    [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
    • [object Object] supports only the INT8 quantization data type.
    • [object Object] only supports the non-transposed mode and supports INT8, INT4, and INT32 data types. In ND format, the shape is {(E, K, N)}. In NZ format, the shape is {(E, N/32, K/16, 16, 32)} for the INT8 data type, {(E, N/64, K/16, 16, 64)} for INT4, and {(E, N/64, K/16, 16, 8)} for INT32.
    • [object Object] supports FLOAT, FLOAT16, and BFLOAT16 data types in the A8W8 scenario, where the shape must be 2D, represented as {(E, N)}. In the A8W4 scenario, it supports the UINT64 data type, and the shape can be 2D or 3D (shape {(E, N)} for per-channel mode and shape {(E, KGroupCount, N)} for per-group mode).
    • The [object Object] parameter is supported. The value [object Object] indicates per-token mode for the activation matrix and per-channel mode for the weight matrix. The value [object Object] indicates per-token mode for the activation matrix and per-group mode for the weight matrix.
    • The [object Object] parameter is not supported.
    • The [object Object] parameter is not supported.
    • In the A8W8 or A8W4 scenario, the length of the N-axis cannot exceed 10240.
    • In the A8W8 scenario, the length of the last axis of [object Object] cannot be greater than or equal to 65536.
    • In the A8W4 scenario, the length of the last axis of [object Object] cannot be greater than or equal to 20000.
    • The data type of [object Object] must be INT8, and the shape can be 2D, for example, (M, N/2).
    • The data type of [object Object] must be FLOAT, and the shape can be 1D, for example, (M,).

• Return

  [object Object] status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return

  [object Object] status code. For details, see .

[object Object]

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • In A8W8/A8W4 quantization scenarios, the following constraints must be met:
    • Data type requirements
    [object Object]
    • In the A8W8 scenario, the length of the N-axis cannot exceed 10240, and the length of the last axis of [object Object] cannot be greater than or equal to 65536.
    • In the A8W4 scenario, the length of the N-axis cannot exceed 10240, and the length of the last axis of [object Object] cannot be greater than or equal to 20000.

• Deterministic computation:

  • [object Object] defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]