[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs per-token symmetric/asymmetric dynamic quantization on the input tensor. In the MOE scenario, smoothScalesOptional for each expert is different and is distinguished by the input groupIndexOptional.

• Formula:

  • Symmetric quantization:

    • When smoothScalesOptional is not provided:

      $$scaleOut=row\_max(abs(x))/127$$ $$yOut=round(x/scaleOut)$$

    • When smoothScalesOptional is provided:

      $$input = x\cdot smoothScalesOptional$$ $$scaleOut=row\_max(abs(input))/127$$ $$yOut=round(input/scaleOut)$$

  • Asymmetric quantization:

    • When smoothScalesOptional is not provided:

      $$scaleOut=(row\_max(x) - row\_min(x))/scale\_opt$$ $$offset=offset\_opt-row\_max(x)/scaleOut$$ $$yOut=round(x/scaleOut+offset)$$

    • When smoothScalesOptional is provided:

      $$input = x\cdot smoothScalesOptional$$ $$scaleOut=(row\_max(input) - row\_min(input))/scale\_opt$$ $$offset=offset\_opt-row\_max(input)/scaleOut$$ $$yOut=round(input/scaleOut+offset)$$

  Where row_max denotes computing the maximum value for each row, and row_min denotes computing the minimum value for each row. When the output yOut is of the INT8 type, scale_opt is 255.0 and offset_opt is 127.0. When the output yOut is of the INT4 or INT32 type, scale_opt is 15.0 and offset_opt is 7.0.

[object Object]

Each operator has calls. First, aclnnDynamicQuantV2GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, aclnnDynamicQuantV2 is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    The data type of the output parameter [object Object] can only be INT4, INT8, or INT32.

  • [object Object]Atlas inference products[object Object] and [object Object]Atlas training products[object Object]:

    • Only FLOAT16 is supported for the input parameter [object Object].
    • The input parameters [object Object] and [object Object] are reserved and are not involved in the calculation in the current version.
    • The input parameter [object Object] must be set to 2.
    • Only INT8 is supported for the output parameter [object Object].

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950 PR/Ascend 950 DT:
  • When the data type of yOut is INT4, the last dimensions of x and yOut must be divisible by 2.
  • When the data type of yOut is INT32, the last dimension of x must be divisible by 8.
• [object Object]Atlas inference products[object Object] and [object Object]Atlas training products[object Object]: The last dimension of the input x must be exactly divisible by 32. Currently, only symmetric quantization is supported, and the BFLOAT16 data type is not supported.
• Deterministic computation:
  • aclnnDynamicQuantV2 defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .