[object Object][object Object][object Object]undefined[object Object]

• Description: Performs input matrix multiplication in a fake-quantization scenario and implements output quantization. Compared with the aclnnWeightQuantBatchMatmulV2 API, this API has the following changes:

  The innerPrecise parameter is added to support the selection of high-precision or high-performance compute mode. In the A16W4 per_group scenario, this parameter can be set to 1 when batchSize <= 16 to improve performance.

• Formula:

  $$y = x @ ANTIQUANT(weight) + bias$$

  In the formula, $weight$ is the input of the fake-quantization scenario, and the dequantization formula $ANTIQUANT(weight)$ is as follows:

  $$ANTIQUANT(weight) = (weight + antiquantOffset) * antiquantScale$$

  When quantScaleOptional is configured, the output is quantized using the following formula:

  $$\begin{aligned} y &= QUANT(x @ ANTIQUANT(weight) + bias) \\ &= (x @ ANTIQUANT(weight) + bias) * quantScale + quantOffset \\ \end{aligned}$$

  If quantScaleOptional is set to nullptr, the out is as follows:

  $$y = x @ ANTIQUANT(weight) + bias$$

[object Object]

Each operator has calls. First, aclnnWeightQuantBatchMatmulV3GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnWeightQuantBatchMatmulV3 is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • Ascend 950PR/Ascend 950DT:

    • The superscript "1" in the data type column of the preceding table indicates that the data type is not supported by this series.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The superscript "2" in the "Data Type" column of the table above indicates data types that are not supported by the products.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

[object Object][object Object]

[object Object]

• Deterministic description: The non-deterministic implementation is used by default. You can enable deterministic implementation by calling aclrtCtxSetSysParamOpt.

• Common constraints

  • When the weight data format is FRACTAL_NZ and the data type is INT4 or INT32, or when the weight data format is ND and the data type is INT32, this parameter is supported only in the INT4Pack scenario. The aclnnConvertWeightToINT4Pack API needs to be used for INT32-to-INT4Pack conversion and ND-to-FRACTAL_NZ conversion. [object Object]See details[object Object]. If the data type is INT4, the inner axis of weight must be an even number.
  • For different fake-quantization algorithm modes, the weight data format FRACTAL_NZ is supported only in the following scenarios:
    • per_channel mode: The weight data type is INT8, and the y data type is not INT8. The weight data type is INT4 or INT32, weight is transposed, and the y data type is not INT8.
    • per_group mode: The weight data type is INT4/INT32, weight and x are not transposed, antiquantGroupSize is 64 or 128, k is antiquantGroupSize aligned, n is 64-aligned, and the y data type is not INT8.

• Performance Optimization Suggestions

  • per_channel mode: To improve performance, you are advised to use the weight input after transpose. If the value range of m is [65, 96], antiquantScale of the UINT64/INT64 data type is recommended.
  • per_group mode: In the A16W4 scenario where batchSize is less than or equal to 16, you can set innerPrecise to 1 and set the weight data format to FRACTAL_NZ to improve performance, but the accuracy may drop.

[object Object]

[object Object][object Object]

[object Object]

• Deterministic description: The default deterministic implementation is used.

[object Object][object Object]

• Common constraints

  • The sizes of m, k, and n in the [object Object] and [object Object] matrices are in the range of [1, 2^31 – 1]. The Reduce dimension k of [object Object] must be the same as that of [object Object].
  • The following quantization modes are supported: pertensor , perchannel , pergroup , and mx .
  • [object Object] does not support transposition. Therefore, non-continuous tensors (../common/non_continuous_tensor.md) are not supported. weight supports discontinuous tensors only in the transposition scenario. The discontinuous tensors of antiquantScale and antiquantOffsetOptional support only the transposition scenario, and the continuity requirements must be the same as those of weight.
  • The shapes supported by different quantization modes of [object Object] are as follows:
    • pertensor : (1,) or (1, 1).
    • perchannel : The input shape is (1, n) or (n,).
    • pergroup : The input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.
    • mx : The input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped. Only 32 is supported.
  • [object Object] and [object Object] are reserved parameters and are not used currently. Null pointers are always passed.

  [object Object][object Object]

  [object Object]
  • Input and output data type combinations
  [object Object]undefined[object Object]

  [object Object][object Object]

  [object Object]
  • Input and Output Data Type Combinations
  [object Object]undefined

  • Constraints

    In addition to the common restrictions, the restrictions in the A16W4 scenario are as follows:

    • If the [object Object] data type is INT4 or FLOAT4_E2M1, the last dimension of the weight must be 2-aligned. If the [object Object] data type is INT32 or FLOAT, the last dimension of the weight must be 8-aligned.
    • If the [object Object] data type is INT32 or FLOAT, the [object Object] API must be used to convert the data from INT32 or FLOAT to tightly packed INT4 or FLOAT4_E2M1. .

[object Object][object Object]

• Performance Optimization Suggestions

  • pertensor : When is ND, the transposed [object Object] input is recommended.
  • perchannel : When the is ND, the transposed [object Object] input is recommended.
  • pergroup and mx : The non-transposed [object Object] input is recommended.
  [object Object]

[object Object][object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]and Ascend 950PR/Ascend 950DT: A16W8 calling example.

  [object Object]

• Ascend 950PR/Ascend 950DT: A16MxFp4 calling example.

  [object Object]