[object Object]

[object Object][object Object]undefined[object Object]

• Description: Adds quantization after the SwiGlu activation function to implement SwiGluQuant computation of the input x. The int8 or int4 quantization output is supported.

• API function differences: Compared with aclnnSwiGluQuant, aclnnSwiGluQuantV2 supports the cumsum and count modes for the passed groupIndexOptional. The modes can be controlled using groupListType. aclnnSwiGluQuantV2 supports the non-MoE scenario (groupIndexOptional is passed as null). aclnnSwiGluQuantV2 also supports the int8 or int4 quantization output yOut, which indicates that the quantization output data type can be controlled using dstType.

• Operator support: Currently, SwiGluQuant supports the MoE scenario (groupIndexOptional is passed) and non-MoE scenario (groupIndexOptional is passed as null). . The input x and group_index of SwiGluQuant come from the outputs of the GroupedMatMul operator and MoeInitRouting. group_index is used to implement dynamic MoE group quantization, static per_tensor quantization, and static per_channel quantization.

• Dynamic quantization formula in the MoE scenario:

  $$Act = SwiGlu(x) = Swish(A)*B \\ Y_{tmp}^0 = Act[0\colon g[0],\colon] * smooth\_scales[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))$$ $$Y = Cast(Mul(Y_{tmp}, Scale))$$

  A indicates the first half of the input x, B indicates the second half of the input x, g indicates group_index, and G indicates the number of groups of group_index. During int8 quantization, $dstTypeScale = 127$ (127 is the maximum value of int8). During int4 quantization, $dstTypeScale = 7$ (7 is the maximum value of int4).

• Static quantization formula in the MoE scenario:

  $$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}^0 = Act(0\colon g[0],\colon) * smooth\_scales[0\colon g[0],\colon] + offsets[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon] + offsets[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\$$ $$Y = Cast(Y_{tmp})$$

  A indicates the first half of the input x, B indicates the second half of the input x, g indicates group_index, and G indicates the number of groups of group_index.

• Dynamic quantization formula in the non-MoE scenario (groupIndexOptional is passed as null):

  $$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act* smooth\_scales(0,\colon)\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))$$ $$Y = Cast(Mul(Y_{tmp}, Scale))$$

  A indicates the first half of the input x, and B indicates the second half of the input x. During int8 quantization, $dstTypeScale = 127$ (127 is the maximum value of int8). During int4 quantization, $dstTypeScale = 7$ (7 is the maximum value of int4).

• Static quantization formula in the non-MoE scenario (groupIndexOptional is passed as null):

  $$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act * smooth\_scales(0,\colon) + offsets(0,\colon) \\$$ $$Y = Cast(Y_{tmp})$$

  A indicates the first half of the input x, and B indicates the second half of the input x.

[object Object]

Each operator has calls. First, aclnnSwiGluQuantV2GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnSwiGluQuantV2 is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic compute:
  • aclnnSwiGluQuantV2 defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .