[object Object][object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas.

• Formulas:

  • Quantization scenario (A8W8; [object Object] represents the activation matrix (left matrix), [object Object] represents the weight matrix (right matrix), and [object Object] represents the INT8 data type):

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: input matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the input matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of prefix sums.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

        [object Object]

      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ BroadCast} \odot w\_scale_{i\ BroadCast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

  ---

  • MSD scenario (A8W4; [object Object] represents the activation matrix (left matrix), [object Object] represents the weight matrix (right matrix), and [object Object] represents the INT4 data type):
    • Definition
      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
    • Input
      • $X∈\mathbb{Z_8}^{M \times K}$: input matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $bias∈\mathbb{R}^{E \times N}$: auxiliary matrix for matrix multiplication (the computation process for generating the auxiliary matrix is described below).
      • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the input matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of prefix sums.
    • Output
      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
    • Computation process
      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. Compute the auxiliary matrix ([object Object]). (Note that the bias computation is performed offline and provided as an input, rather than being executed within the operator.)

        • For per-channel quantization ($w\_scale$ is 2D):

          $bias_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$

        • For per-group quantization ($w\_scale$ is 3D):

          $bias_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$

          Note: $num\_per\_group = K // K\_group\_num$

      • 3. Perform the following computation based on the input parameters determined by grouping:

        • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

        • 3.2. Enable per-channel or per-group quantization during matrix multiplication.

          Per-channel:

          $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          Per-group:

          $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

          $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

          $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + bias_{i}) \odot x\_scale_{i}$

          $C_{i,act}, gate_{i} = split(C_{i})$

          $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  • x (aclTensor*, computation input): left matrix, $X$ in the formula, aclTensor on the device. The shape supports 2D. Assuming the shape is [M,K], K must be less than 65536. The data type can be INT8, and the can be ND. are supported.

  • weight (aclTensor*, computation input): weight matrix, $W$ in the formula, aclTensor on the device. The data type can be INT8, INT4, or INT32 (INT32 is used for adaptation. Actually, one INT32 data record is interpreted as eight INT4 data records). are supported.

    • A8W4 scenario: The shape can be 3D or 5D, corresponding to the ND or FRACTAL_NZ, respectively.
    • A8W8 scenario: The shape can be 5D, and the can be FRACTAL_NZ.

  • bias (aclTensor*, computation input): auxiliary matrix for matrix multiplication, $bias$ in the formula. The shape supports 2D, and the data type can be FP32. This parameter is only effective in the A8W4 scenario and needs to be passed as a null pointer in the A8W8.

  • offset (aclTensor*, computation input): offset of per-channel asymmetric dequantization, $offset$ in the formula. The shape supports two dimensions. The data type is float. This input is reserved and is not supported currently. You need to pass a null pointer.

  • weightScale (aclTensor*, computation input): quantization factor of the right matrix, $w\_scale$ in the formula, aclTensor on the device. The length of the first axis must be the same as the first axis of [object Object]. The length of the last axis must be the same as the last axis of [object Object] restored to the ND format. The can be ND. are supported.

    • A8W4 scenario: The shape can be 2D or 3D, and the data type can be UINT64.
    • A8W8 scenario: The shape can be 2D, and the data type can be FLOAT, FLOAT16, or BFLOAT16.

  • xScale (aclTensor*, computation input): quantization factor of the left matrix, $x\_scale$ in the formula, aclTensor on the device. The shape supports 1D. The length must be the same as the first axis of [object Object]. The data type can be FLOAT. The can be ND. are supported.

  • groupList (aclTensor*, computation input): number of tokens involved in the computation of each group, $grouplist$ in the formula, aclTensor on the device. The shape supports 1D. The length must be the same as the first axis of [object Object]. The data type can be INT64. The can be ND. are supported. The last value in [object Object] constrains the valid portion of the output data. For details, see the computation process.

  • output (aclTensor*, computation output): quantization result, $Q$ in the formula, aclTensor on the device. The data type can be INT8, and the shape can be 2D. The can be ND. are supported.

  • outputScale (aclTensor*, computation output): quantization factor, $Q\_scale$ in the formula, aclTensor on the device. The data type can be FLOAT, and the shape can be 1D. The can be ND. are supported.

  • outputOffset (aclTensor*, computation output): offset of asymmetric quantization, $Q\_offset$ in the formula, aclTensor on the device. The shape can be 1D, and the data type can be FLOAT. This input is reserved and is not supported currently. You need to pass a null pointer.

  • workspaceSize (uint64_t*, output parameter): size of the workspace to be allocated on the NPU device.

  • executor (aclOpExecutor**, computation output): operator executor, containing the operator computation process.

• Return

  [object Object] status code. For details, see .

[object Object]

• Parameters

  • workspace (void*, input parameter): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input parameter): workspace size to be allocated on the device, which is obtained by the first-phase API aclnnGroupedMatmulSwigluQuantGetWorkspaceSize.
  • executor (aclOpExecutor*, input parameter): operator executor, containing the operator computation process.
  • stream (aclrtStream, input parameter): stream for executing the task.

• Return

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.

[object Object]

• 1. The length of the last axis of [object Object] cannot be greater than or equal to 65536.
• 2. The data format of [object Object] must be FRACTAL_NZ.
• 3. The length of the N-axis cannot exceed 10240.

[object Object][object Object]

• 1. The length of the last axis of [object Object] cannot be greater than or equal to 20000.
• 2. The length of the N-axis cannot exceed 10240.

[object Object][object Object]

• Single-aclnn-operator calling

The following single-aclnn-operator calling example is for reference only. For details, see .