[object Object]

[object Object][object Object]undefined[object Object]

• Function: performs matrix multiplication in the fake-quantization scenario and quantizes the output.
• Formula:$$y = x @ ANTIQUANT(weight) + bias$$ In the formula, $weight$ is the input of the fake-quantization scenario, and the dequantization formula $ANTIQUANT(weight)$ is as follows:$$ANTIQUANT(weight) = (weight + antiquantOffset) * antiquantScale$$
  • If the output does not need to be quantized, the calculation formula is as follows:
  $$y = x @ ANTIQUANT(weight) + bias$$
  • If the output needs to be quantized, the quantization formula is as follows:
  $$\begin{aligned} y &= QUANT(x @ ANTIQUANT(weight) + bias) \\ &= (x @ ANTIQUANT(weight) + bias) * quantScale + quantOffset \\ \end{aligned}$$

[object Object]

Each operator has calls. First, aclnnWeightQuantBatchMatmulV2GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnWeightQuantBatchMatmulV2 is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • Ascend 950PR/Ascend 950DT:

    • The superscript "1" in the data type column of the preceding table indicates that the data type is not supported by this series.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The superscript "2" in the "Data Type" column of the table above indicates data types that are not supported by the products.

  • [object Object]Atlas inference products[object Object]

    • The superscript "3" in the data type column of the preceding table indicates that the data type is not supported by this series.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

[object Object][object Object]

[object Object]

• Deterministic description: The default implementation is non-deterministic. You can enable deterministic implementation by calling aclrtCtxSetSysParamOpt.

• [object Object] (aclTensor *, input for computation): When the matrix is not transposed, the value of m is within the range of [1, 2^31 – 1]. When the matrix is transposed, the value of m is within the range of [1, 65535].

• [object Object] (aclTensor *, input for computation): The dimension can be 2D. The dimension k of Reduce must be the same as that of [object Object]. The data type can be INT8, INT4, or INT32. When the [object Object] is FRACTAL_NZ and the data type is INT4 or INT32, or when the [object Object] is ND and the data type is INT32, this parameter is supported only in the INT4Pack scenario. [object Object] is also used for INT32-to-INT4Pack conversion and ND-to-FRACTAL_NZ conversion. For details, see the . If the data type is INT4, the inner axis of [object Object] must be an even number. are supported only in the transpose scenario. The shape can be (k, n), where k indicates the size of the first dimension of the matrix, and n indicates the size of the second dimension of the matrix. For different fake-quantization algorithm modes, the [object Object] FRACTAL_NZ is supported only in the following scenarios:

  • perchannel :
    • The [object Object] data type is INT8, and the y data type is not INT8.
    • The [object Object] data type is INT4 or INT32, [object Object] is transposed, and the y data type is not INT8.
  • pergroup : The [object Object] data type is INT4 or INT32, [object Object] and [object Object] are not transposed, antiquantGroupSize is 64 or 128, k is a multiple of antiquantGroupSize, n is a multiple of 64, and the y data type is not INT8.

• antiquantScale (aclTensor *, compute input): The supported data types are FLOAT16, BFLOAT16, UINT64, and INT64. When the data type is FLOAT16 or BFLOAT16, the data type must be the same as that of the input x. When the data type is UINT64 or INT64, x supports only FLOAT16 and is not transposed, weight supports only INT8 and is transposed in ND format, and the quantization mode must be perchannel . Null pointers must be passed for quantScaleOptional and quantOffsetOptional. The value of m ranges from 1 to 96. The values of k and n must be multiples of 64. First, the aclnnCast API is used to perform the FLOAT16-to-FLOAT32 conversion. For details, see . Then, the aclnnTransQuantParamV2 API is used to perform the FLOAT32-to-UINT64 conversion. For details, see . are supported only in the transpose scenario. For different fake-quantization algorithm modes, [object Object] supports the following shapes:

  • pertensor : The input shape is (1,) or (1, 1).
  • perchannel : The input shape is (1, n) or (n,).
  • pergroup : The input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.

• [object Object] (aclTensor *, input for computation): The data type can be FLOAT16, BFLOAT16, or INT32. When the data type is FLOAT16 or BFLOAT16, the data type must be the same as that of the input [object Object]. When the data type is INT32, the value range is restricted to [–128, 127]. x supports only FLOAT16, weight supports only INT8, and [object Object] supports only UINT64 or INT64. are supported only in the transpose scenario.

• [object Object] (aclTensor *, input for computation): The data type can be UINT64, and the data format can be ND. are not supported. It is an optional parameter. When it is not required, pass a null pointer to it. For different fake-quantization algorithm modes, the supported shapes are as follows:

  • pertensor : The input shape is (1,) or (1, 1).
  • perchannel : The input shape is (1, n) or (n,).

• [object Object] (aclTensor *, input for computation): The data type can be FLOAT, and the data format can be ND. It is an optional parameter. When it is not required, pass a null pointer to it. If it is required, the shape must be the same as that of [object Object]. are not supported.

• [object Object] (aclTensor *, input for computation): The dimension can be 1 or 2, and the shape can be (n,) or (1, n). The data type can be FLOAT16 or FLOAT. When the data type of [object Object] is BFLOAT16, the data type of this parameter must be FLOAT. When the data type of [object Object] is FLOAT16, the data type of this parameter must be FLOAT16.

• antiquantGroupSize (int, compute input): groupSize input for dequantizing the input weight in pergroup or mx of the fake quantization algorithm. It describes the size of the data to be dequantized corresponding to a group of dequantization parameters in the Reduce direction. If the fake quantization algorithm is not in pergroup or mx , pass 0. If the fake quantization algorithm is pergroup , the value range is [32, k – 1] and the value must be a multiple of 32. In the mx , only 32 is supported.

• [object Object] (aclTensor *, output): The dimension supports 2D, and the shape supports (m, n). The data type can be FLOAT16, BFLOAT16, or INT8. If [object Object] exists, the data type is INT8. If [object Object] does not exist, the data type can be FLOAT16 or BFLOAT16, and must be the same as the data type of the input [object Object].

• Performance optimization suggestions:

  • pertensor : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended.
  • pergroup : The non-transposed weight input is recommended.
  • perchannel : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended. If the value range of m is [65, 96], antiquantScale of the UINT64 or INT64 data type is recommended.

[object Object]

[object Object][object Object]

[object Object]

• Deterministic description: The non-deterministic implementation is used by default. You can enable the deterministic implementation by calling aclrtCtxSetSysParamOpt.

• [object Object] (aclTensor, input for computation): The data type is FLOAT16. The shape supports 2 to 6 dimensions. The input shape must be (batch, m, k), where batch indicates the batch size of the matrix and supports 0 to 4 dimensions, m indicates the size of the first dimension of the single-batch matrix, and k indicates the size of the second dimension of the single-batch matrix. The batch dimension must meet the broadcast relationship with the batch dimension of [object Object]. When the fake-quantization algorithm is in per-tensor mode (../common/quant_mode_introduction.md), the value of m x k cannot exceed 512000000.

• [object Object] (aclTensor *, input for computation): The shape supports 2 to 6 dimensions. The batch dimension must meet the broadcast relationship with the batch dimension of [object Object]. The data type is INT8. Details are as follows:

  • If the is ND, the input shape must be (batch, k, n), where batch indicates the batch size of the matrix and can be zero- to four-dimensional, k indicates the size of the first dimension of the single batch matrix, and n indicates the size of the second dimension of the single batch matrix.
  • If the is FRACTAL_NZ:
    • The input shape must be (batch, n, k), where batch indicates the batch size of the matrix and can be zero- to four-dimensional, k indicates the size of the first dimension of the single batch matrix, and n indicates the size of the second dimension of the single batch matrix.
    • This API is used together with aclnnCalculateMatmulWeightSizeV2 and aclnnTransMatmulWeight to convert the input format from ND to FRACTAL_NZ.

• [object Object] (aclTensor *, input for computation): The data type is FLOAT16. The data type must be the same as that of the input [object Object]. For different fake-quantization algorithm modes, [object Object] supports the following shapes:

  • pertensor : The input shape is (1,) or (1, 1).
  • perchannel : The input shape is (n, 1) or (n, ). are not supported.
  • pergroup : The input shape is related to the data format of [object Object] as follows:
    • When the data format of [object Object] is ND, the input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.
    • When the data format of [object Object] is FRACTAL_NZ, the input shape is (n, ⌈k/group_size⌉), where group_size indicates the size of each group to which k is to be grouped.

• [object Object] (aclTensor *, input for computation): The data type is FLOAT16. The data type must be the same as that of the input [object Object].

• [object Object] (aclTensor *, input for computation): reserved. Currently, this parameter is not used and a null pointer is always passed.

• [object Object] (aclTensor *, input for computation): reserved. Currently, this parameter is not used and a null pointer is always passed.

• [object Object] (aclTensor *, input for computation): The data type is FLOAT16. One to six dimensions are supported. When batch is used, the input shape must be (batch, 1, n), where batch must be the same as the batch after the batch dimensions of x and weight are broadcast. When batch is not used, the input shape must be (n,) or (1, n).

• [object Object] (int, input for computation): The data type is FLOAT16. Two to six dimensions are supported, and the shape can be (batch, m, n), where batch is optional. The batch dimensions of x and weight can be broadcast. The output batch is the same as the broadcast batch. m and n are the same as m of x and n of weight, respectively.

• [object Object] (aclTensor *, output for computation):

[object Object]

[object Object][object Object]

[object Object]

• Deterministic description: The default deterministic implementation is used.

[object Object][object Object]

• Common constraints

  • The sizes of the [object Object] and [object Object] matrices m, k, and n are in the range of [1, 2^31 – 1]. The dimension k of [object Object]Reduce must be the same as that of [object Object].
  • The following quantization modes are supported: pertensor , perchannel , pergroup , and mx .
  • [object Object] does not support transposition. Therefore, are not supported. The weight supports discontinuous tensors only in the transposition scenario. The anti-quantization scale and anti-quantization offset optional support discontinuous tensors only in the transposition scenario, and the continuity requirements must be the same as those of the weight.
  • The shapes supported by different quantization modes of [object Object] are as follows:
    • pertensor : (1,) or (1, 1).
    • perchannel : The input shape is (1, n) or (n,).
    • pergroup : The input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.
    • mx [quantization mode](../common/quantization introduction.md): The input shape is (⌈k/group_size ⌉, n), where group_size indicates the size of each group to be grouped. The value can only be 32.
  • [object Object] and [object Object] are reserved parameters and are not used currently. Empty pointers are always passed.

  [object Object][object Object]

  [object Object]
  • Input and Output Data Type Combinations
  [object Object]undefined[object Object]

  [object Object][object Object]

  [object Object]
  • Input and Output Data Type Combinations
  [object Object]undefined

  • Constraints

    In addition to the common restrictions, the restrictions for the A16W4 scenario are as follows:

    • If the [object Object] data type is INT4 or FLOAT4_E2M1, the last dimension of the weight must be 2-aligned. If the [object Object] data type is INT32 or FLOAT, the last dimension of the weight must be 8-aligned.
    • If the [object Object] data type is INT32 or FLOAT, the [object Object] API must be used to convert the data from INT32 or FLOAT to tightly packed INT4 or FLOAT4_E2M1. .
    • The weight FRACTAL_NZ is supported only in the following scenarios:
      • perchannel : The [object Object] data type is INT4 or INT32, [object Object] is not transposed, and [object Object] is not transposed.
      • pergroup : The [object Object] data type is INT4, INT32, FLOAT4_E2M1, or FLOAT, [object Object] is not transposed, and [object Object] is not transposed.
      • mx : The [object Object] data type is FLOAT4_E2M1 or FLOAT, [object Object] is not transposed, and [object Object] is not transposed.

[object Object][object Object]

• Performance Optimization Suggestions

  • pertensor : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended.
  • perchannel : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended.
  • pergroup and mx : The non-transposed [object Object] input is recommended.
  [object Object]

[object Object][object Object]

The following example is for reference only. For details, see .

• A16W8 calling example:

  [object Object]

• The following is an example of invoking the A16W4 interface. The [object Object] interface needs to be invoked to assist in the invocation.

  [object Object]