[object Object]

[object Object][object Object]undefined[object Object]

• Description: Adds quantization after the SwiGlu activation function to implement SwiGluQuant computation of the input x.

• Operator support: Currently, SwiGluQuant only supports the MoE scenario. The input x and group_index of SwiGluQuant come from the outputs of the GroupedMatMul operator and MoeInitRouting. group_index is used to implement dynamic MoE group quantization, static per_tensor quantization, and static per_channel quantization.

• Dynamic quantization formula:

  $$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}[0\colon g[0],\colon] = Act[0\colon g[0],\colon] * smooth\_scales[0,\colon], i=0 \\ Y_{tmp}[g[i]\colon g[i+1], \colon] = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[i+1, \colon], i \in (0, G) \cap \mathbb{Z}\\ scale=127/row\_max(abs(Y_{tmp}))$$ $$Y = Cast(Mul(Y_{tmp}, Scale))$$

  A indicates the first half of the input x, B indicates the second half of the input x, g indicates group_index, and G indicates the number of groups of group_index.

• Static quantization formula:

  $$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}[0\colon g[0],\colon] = Act[0\colon g[0],\colon] * smooth\_scales[0,\colon] + offsets[0,\colon], i=0 \\ Y_{tmp}[g[i]\colon g[i+1], \colon] = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[i+1, \colon] + offsets[i+1, \colon], i \in (0, G) \cap \mathbb{Z}\\$$ $$Y = Cast(Y_{tmp})$$

  A indicates the first half of the input x, B indicates the second half of the input x, g indicates group_index, and G indicates the number of groups of group_index.

[object Object]

Each operator has calls. First, aclnnSwiGluQuantGetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnSwiGluQuant is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns: [object Object]: status code. For details, see .

[object Object]

• Deterministic compute:
  • aclnnSwiGluQuant defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .