[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs per-token symmetric/asymmetric dynamic quantization on the input tensor. In the MOE scenario, smoothScalesOptional for each expert is different and is distinguished by the input groupIndexOptional.

• Formula:

  • Symmetric quantization:

    • When smoothScalesOptional is not provided:

      $$scaleOut=row\_max(abs(x))/127$$ $$yOut=round(x/scaleOut)$$

    • When smoothScalesOptional is provided:

      $$input = x\cdot smoothScalesOptional$$ $$scaleOut=row\_max(abs(input))/127$$ $$yOut=round(input/scaleOut)$$

  • Asymmetric quantization:

    • When smoothScalesOptional is not provided:

      $$scaleOut=(row\_max(x) - row\_min(x))/scale\_opt$$ $$offset=offset\_opt-row\_max(x)/scaleOut$$ $$yOut=round(x/scaleOut+offset)$$

    • When smoothScalesOptional is provided:

      $$input = x\cdot smoothScalesOptional$$ $$scaleOut=(row\_max(input) - row\_min(input))/scale\_opt$$ $$offset=offset\_opt-row\_max(input)/scaleOut$$ $$yOut=round(input/scaleOut+offset)$$

  Where row_max denotes computing the maximum value for each row, and row_min denotes computing the minimum value for each row. When the type of the output yOut is INT8, scale_opt is 255.0 and offset_opt is 127.0; when the type of yOut is INT4, scale_opt is 15.0 and offset_opt is 7.0.

[object Object]

Each operator has calls. First, aclnnDynamicQuantV2GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, aclnnDynamicQuantV2 is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas inference series products[object Object] and [object Object]Atlas training series products[object Object]:
    • Only FLOAT16 is supported for the input parameter [object Object].
    • The input parameters [object Object] and [object Object] must be nullptr.
    • The input parameter [object Object] must be set to 2.
    • Only INT8 is supported for the output parameter [object Object].

• Returns

  aclnnStatus: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  aclnnStatus: status code. For details, see .

[object Object]

• When the data type of [object Object] is INT4, the last dimensions of [object Object] and [object Object] must be divisible by 2.
• When the data type of [object Object] is INT32, the last dimension of [object Object] must be divisible by 8.
• [object Object]Atlas inference series products[object Object]: Only 32-bit aligned data is supported on the tail axis. Currently only symmetric quantization is supported, and the BFLOAT16 data type is not supported.
• Deterministic computation:
  • aclnnDynamicQuantV2 defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .