[object Object][object Object][object Object]undefined[object Object]

• Description: Fuses [object Object], [object Object], [object Object], and [object Object]. For details, see the formulas. This API is the weightNZ specialization version of .

• Formulas:

  • Quantization scenario (A8W8):

    • Definition

      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

    • Input

      • $X∈\mathbb{Z_8}^{M \times K}$: input matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_8}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the input matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of prefix sums.

    • Output

      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.

    • Computation process

      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.

        [object Object]

      • 2. Perform the following computation based on the input parameters determined by grouping:

        $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ BroadCast} \odot w\_scale_{i\ BroadCast}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

  ---

  • MSD scenario (A8W4):
    • Definition
      • ⋅ indicates matrix multiplication.
      • ⊙ indicates element-wise multiplication.
      • $\left \lfloor x\right \rceil$ indicates rounding [object Object] to the nearest integer.
      • $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
      • $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
      • $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
    • Input
      • $X∈\mathbb{Z_8}^{M \times K}$: input matrix (left matrix), where [object Object] indicates the total number of tokens and [object Object] indicates the feature dimension.
      • $W∈\mathbb{Z_4}^{E \times K \times N}$: grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the feature dimension, and [object Object] indicates the output dimension.
      • $bias∈\mathbb{R}^{E \times N}$: auxiliary matrix for matrix multiplication (the computation process for generating the auxiliary matrix is described below).
      • $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$: per-channel scale factor for the grouped weight matrix (right matrix), where [object Object] indicates the number of experts, [object Object] indicates the number of groups along the K-axis, and [object Object] indicates the output dimension.
      • $x\_scale∈\mathbb{R}^{M}$: per-token scale factor for the input matrix (left matrix), where [object Object] indicates the total number of tokens.
      • $grouplist∈\mathbb{N}^{E}$: grouped index list of prefix sums.
    • Output
      • $Q∈\mathbb{Z_8}^{M \times N / 2}$: quantized output matrix.
      • $Q\_scale∈\mathbb{R}^{M}$: quantization scale factor.
    • Computation process
      • 1. Determine the tokens of the current group based on [object Object], where $i \in [0,Len(groupList)]$.
        • The grouping logic is the same as that of A8W8.
      • 2. Compute the auxiliary matrix ([object Object]). (Note that the bias computation is performed offline and provided as an input, rather than being executed within the operator.)

        • For per-channel quantization ($w\_scale$ is 2D):

          $bias_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$

        • For per-group quantization ($w\_scale$ is 3D):

          $bias_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$

          Note: $num\_per\_group = K // K\_group\_num$

      • 3. Perform the following computation based on the input parameters determined by grouping:

        • 3.1. Convert the left matrix $\mathbb{Z_8}$ into two $\mathbb{Z_4}$ components that represent the high and low bits. $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

        • 3.2. Enable per-channel or per-group quantization during matrix multiplication.

          Per-channel:

          $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

          Per-group:

          $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

          $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        • 3.3. Restore the matrix multiplication results of the high and low bits into the overall result.

          $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + bias_{i}) \odot x\_scale_{i}$

          $C_{i,act}, gate_{i} = split(C_{i})$

          $S_{i}=Swish(C_{i,act})\odot gate_{i}$    where $Swish(x)=\frac{x}{1+e^{-x}}$

      • 3. Quantize the output.

        $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

        $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  • x (aclTensor*, computation input): left matrix, $X$ in the formula, aclTensor on the device. The shape supports 2D. Assuming the shape is [M,K], K must be less than 65536. The data type can be INT8, and the can be ND. are supported.

  • weight (aclTensor*, computation input): weight matrix, $W$ in the formula, aclTensor on the device. The shape supports 5D. The data type can be INT8, INT4, or INT32 (INT32 is used for adaptation. Actually, one INT32 data record is interpreted as eight INT4 data records). The can be FRACTAL_NZ. are supported. Note that this API ignores the weight data format and considers it as the FRACTAL_NZ format.

  • bias (aclTensor*, computation input): matrix multiplication bias, $bias$ in the formula. The shape supports 2D. The data type can be INT32. This input is reserved and is not supported currently. You need to pass a null pointer.

  • offset (aclTensor*, computation input): offset of per-channel asymmetric dequantization, $offset$ in the formula. The shape supports 2D. The data type is float. This input is reserved and is not supported currently. You need to pass a null pointer.

  • weightScale (aclTensor*, computation input): quantization factor of the right matrix, $w\_scale$ in the formula, aclTensor on the device. The shape supports 2D. The length of the first axis must be the same as the first axis of [object Object]. The length of the last axis must be the same as the last axis of [object Object] restored to the ND format. The data type can be FLOAT, FLOAT16, or BFLOAT16. The can be ND. are supported.

  • xScale (aclTensor*, computation input): quantization factor of the left matrix, $x\_scale$ in the formula, aclTensor on the device. The shape supports 1D. The length must be the same as the first axis of [object Object]. The data type can be FLOAT. The can be ND. are supported.

  • groupList (aclTensor*, computation input): number of tokens involved in the computation of each group, $grouplist$ in the formula, aclTensor on the device. The shape supports 1D. The length must be the same as the first axis of [object Object]. The data type can be INT64. The can be ND. are supported. The last value in [object Object] constrains the valid portion of the output data. For details, see the computation process.

  • output (aclTensor*, computation output): quantization result, $Q$ in the formula, aclTensor on the device. The data type can be INT8, and the shape can be 2D. The can be ND. are supported.

  • outputScale (aclTensor*, computation output): quantization factor, $Q\_scale$ in the formula, aclTensor on the device. The data type can be FLOAT, and the shape can be 1D. The can be ND. are supported.

  • outputOffset (aclTensor*, computation output): offset of asymmetric quantization, $Q\_offset$ in the formula, aclTensor on the device. The shape can be 1D, and the data type can be FLOAT. This input is reserved and is not supported currently. You need to pass a null pointer.

  • workspaceSize (uint64_t*, output parameter): size of the workspace to be allocated on the NPU device.

  • executor (aclOpExecutor**, computation output): operator executor, containing the operator computation process.

• Return

  [object Object] status code. For details, see .

[object Object]

• Parameters

  • workspace (void*, input parameter): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input parameter): workspace size to be allocated on the device, which is obtained by the first-phase API [object Object].
  • executor (aclOpExecutor*, input parameter): operator executor, containing the operator computation process.
  • stream (aclrtStream, input parameter): stream for executing the task.

• Return

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.

[object Object]

• 1. The length of the last axis of [object Object] cannot be greater than or equal to 65536.
• 2. The length of the N-axis cannot exceed 10240.

[object Object][object Object]

• 1. The length of the last axis of [object Object] cannot be greater than or equal to 20000.
• 2. The length of the N-axis cannot exceed 10240.

[object Object][object Object]

The following example is for reference only. For details, see .