[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs input matrix multiplication in a fake-quantization scenario and implements output quantization.
• Formula:$$y = x @ ANTIQUANT(weight) + bias$$ In the formula, $weight$ is the input of the fake-quantization scenario, and the dequantization formula $ANTIQUANT(weight)$ is as follows:$$ANTIQUANT(weight) = (weight + antiquantOffset) * antiquantScale$$ When the output needs to be quantized, the quantization formula is as follows:$$\begin{aligned} y &= QUANT(x @ ANTIQUANT(weight) + bias) \\ &= (x @ ANTIQUANT(weight) + bias) * quantScale + quantOffset \\ \end{aligned}$$ When the output does not need to be quantized, the formula is as follows:$$y = x @ ANTIQUANT(weight) + bias$$

[object Object]

Each operator has calls. First, aclnnWeightQuantBatchMatmulV2GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnWeightQuantBatchMatmulV2 is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  • x (aclTensor *, compute input): left input matrix of matrix multiplication, input [object Object] in the formula, and aclTensor on the device. The can be ND. are supported only in the transpose scenario.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: Two dimensions are supported. The shape can be (m, k), where m indicates the size of the first dimension of the matrix, and k indicates the size of the second dimension of the matrix. The Reduce dimension k must be the same as the Reduce dimension k of [object Object]. The data type can be FLOAT16 or BFLOAT16. When the [object Object] matrix is not transposed, m is in the range of [1, 2^31-1]. When the [object Object] matrix is transposed, m is in the range of [1, 65535].
    • [object Object]Atlas inference series products[object Object]: The data type can be FLOAT16. The shape can be two- to six-dimensional. The input shape must be (batch, m, k), where batch indicates the batch size of the matrix and can be zero- to four-dimensional. m indicates the size of the first dimension of the single batch matrix, and k indicates the size of the second dimension of the single batch matrix. The batch dimension must meet the with the batch dimension of [object Object]. When the fake quantization algorithm mode is pertensor , m × k cannot exceed 512000000.

  • weight (aclTensor *, compute input): right input matrix of matrix multiplication, input [object Object] in the formula, and aclTensor on the device. The can be ND or FRACTAL_NZ.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: Two dimensions are supported. The reduce dimension k must be the same as the reduce dimension k of [object Object]. The data type can be INT8, INT4, or INT32. When the [object Object] is FRACTAL_NZ and the data type is INT4 or INT32, or when the [object Object] is ND and the data type is INT32, this parameter is supported only in the INT4Pack scenario. [object Object] is also used for INT32-to-INT4Pack conversion and ND-to-FRACTAL_NZ conversion. For details, see the . If the data type is INT4, the inner axis of [object Object] must be an even number. are supported only in the transpose scenario. The shape can be (k, n), where k indicates the size of the first dimension of the matrix, and n indicates the size of the second dimension of the matrix. For different fake-quantization algorithm modes, the [object Object] FRACTAL_NZ is supported only in the following scenarios:
      • perchannel :
        • The [object Object] data type is INT8, and the y data type is not INT8.
        • The [object Object] data type is INT4 or INT32, [object Object] is transposed, and the y data type is not INT8.
      • pergroup : The [object Object] data type is INT4 or INT32, [object Object] and [object Object] are not transposed, antiquantGroupSize is 64 or 128, k is a multiple of antiquantGroupSize, n is a multiple of 64, and the y data type is not INT8.
    • [object Object]Atlas inference series products[object Object]: Two to six dimensions are supported. The batch dimension must meet the with the batch dimension of [object Object]. The data type can be INT8. Details are as follows:
      • If the is ND, the input shape must be (batch, k, n), where batch indicates the batch size of the matrix and can be zero- to four-dimensional, k indicates the size of the first dimension of the single batch matrix, and n indicates the size of the second dimension of the single batch matrix.
      • If the is FRACTAL_NZ:
        • The input shape must be (batch, n, k), where batch indicates the batch size of the matrix and can be zero- to four-dimensional, k indicates the size of the first dimension of the single batch matrix, and n indicates the size of the second dimension of the single batch matrix.
        • aclnnCalculateMatmulWeightSizeV2 and aclnnTransMatmulWeight are also used to convert the input format from ND to FRACTAL_NZ. For details, see the .

  • antiquantScale (aclTensor *, compute input): dequantization scale parameter, the input [object Object] in the dequantization formula, and aclTensor on the device. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The supported data types are FLOAT16, BFLOAT16, UINT64, and INT64. When the data type is FLOAT16 or BFLOAT16, the data type must be the same as that of the input [object Object]. When the data type is UINT64 or INT64, [object Object] supports only FLOAT16 and is not transposed, [object Object] supports only INT8 and is transposed in ND format, and the quantization mode must be perchannel . Null pointers must be passed for quantScaleOptional and quantOffsetOptional. The value of m ranges from 1 to 96. The values of k and n must be multiples of 64. First, the aclnnCast API is used to perform the FLOAT16-to-FLOAT32 conversion. For details, see . Then, the aclnnTransQuantParamV2 API is used to perform the FLOAT32-to-UINT64 conversion. For details, see . are supported only in the transpose scenario. For different fake-quantization algorithm modes, [object Object] supports the following shapes:
      • pertensor : The input shape is (1,) or (1, 1).
      • perchannel : The input shape is (1, n) or (n,).
      • pergroup : The input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.
    • [object Object]Atlas inference series products[object Object]: The supported data type is FLOAT16. The data type must be the same as that of the input [object Object]. For different fake-quantization algorithm modes, [object Object] supports the following shapes:
      • pertensor : The input shape is (1,) or (1, 1).
      • perchannel : The input shape is (n, 1) or (n, ). are not supported.
      • pergroup : The input shape is related to the data format of [object Object] as follows:
        • When the data format of [object Object] is ND, the input shape is (⌈k/group_size⌉, n), where group_size indicates the size of each group to which k is to be grouped.
        • When the data format of [object Object] is FRACTAL_NZ, the input shape is (n, ⌈k/group_size⌉), where group_size indicates the size of each group to which k is to be grouped.

  • antiquantOffsetOptional (aclTensor*, compute input): dequantization offset parameter, [object Object] in the dequantization formula, and aclTensor on the device. It is an optional parameter. When it is not required, pass a null pointer to it. When it is required, the shape must be the same as that of [object Object]. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The supported data types are FLOAT16, BFLOAT16, and INT32. When the data type is FLOAT16 or BFLOAT16, it must be the same as the data type of the input [object Object]. When the data type is INT32, the value range is [–128, 127], the data type of x can only be FLOAT16, the data type of weight can only be INT8, and the data type of [object Object] can only be UINT64 or INT64. are supported only in the transpose scenario.
    • [object Object]Atlas inference series products[object Object]: The supported data type is FLOAT16. The data type must be the same as that of the input [object Object].

  • quantScaleOptional (aclTensor *, compute input): quantization parameter, aclTensor on the device, which is converted from the data of [object Object] and [object Object] in the quantization formula through the [object Object] API.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The supported data type is UINT64, and the supported is ND. are not supported. It is an optional parameter. When it is not required, pass a null pointer to it. For different fake-quantization algorithm modes, the supported shapes are as follows:
      • pertensor : The input shape is (1,) or (1, 1).
      • perchannel : The input shape is (1, n) or (n,).
    • [object Object]Atlas inference series products[object Object]: This parameter is reserved and not used currently. It is fixed as a null pointer.

  • quantOffsetOptional (aclTensor*, compute input): quantization offset parameter, [object Object] in the quantization formula, and aclTensor on the device.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The supported data type is FLOAT, and the supported is ND. It is an optional parameter. When it is not required, pass a null pointer to it. If it is required, the shape must be the same as that of [object Object]. are not supported.
    • [object Object]Atlas inference series products[object Object]: This parameter is reserved and not used currently. It is fixed as a null pointer.

  • biasOptional (aclTensor *, compute input): bias input, [object Object] in the formula, and aclTensor on the device. It is an optional parameter. When it is not required, pass a null pointer to it. The can be ND. are not supported.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: One or two dimensions are supported, and the shape (n,) or (1, n) is supported. The data type can be FLOAT16 or FLOAT. When the data type of [object Object] is BFLOAT16, the data type of this parameter must be FLOAT. When the data type of [object Object] is FLOAT16, the data type of this parameter must be FLOAT16.
    • [object Object]Atlas inference series products[object Object]: The data type can be FLOAT16. One to six dimensions are supported. When batch is used, the input shape must be (batch, 1, n), where batch must be the same as the batch after the batch dimensions of x and weight are broadcast. When batch is not used, the input shape must be (n,) or (1, n).

  • antiquantGroupSize (int, compute input): groupSize input for dequantizing the input [object Object] in pergroup or mx of the fake quantization algorithm. It describes the size of the data to be dequantized corresponding to a group of dequantization parameters in the Reduce direction. If the fake quantization algorithm is not in pergroup or mx , pass 0. If the fake quantization algorithm is pergroup , the value range is [32, k – 1] and the value must be a multiple of 32. In the mx , only 32 is supported.

  • y (aclTensor*, compute output): compute output, [object Object] in the formula, aclTensor on the device. The can be ND. are not supported.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: Two dimensions are supported, and shape (m, n) is supported. The data type can be FLOAT16, BFLOAT16, or INT8. If [object Object] exists, the data type is INT8. If [object Object] does not exist, the data type can be FLOAT16 or BFLOAT16, and must be the same as the data type of the input [object Object].
    • [object Object]Atlas inference series products[object Object]: The data type can be FLOAT16. Two to six dimensions are supported, and the shape can be (batch, m, n), where batch is optional. The batch dimensions of x and weight can be broadcast. The output batch is the same as the broadcast batch. m and n are the same as m of x and n of weight, respectively.

  • workspaceSize (uint64_t *, output): size of the workspace required to be allocated on the device.

  • executor (aclOpExecutor **, output): operator executor, containing the operator computation process.

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Parameters

  • workspace (void *, input): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by the first-phase API [object Object].
  • executor (aclOpExecutor *, input): operator executor, containing the operator computation process.
  • stream (aclrtStream, input): stream for executing the task.

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Deterministic description:
  • [object Object]Atlas training series products[object Object] and [object Object]Atlas inference series products[object Object]: aclnnWeightQuantBatchMatmulV2 defaults to a non-deterministic implementation. You can call aclrtCtxSetSysParamOpt to enable deterministic compute.

Performance optimization suggestions:

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:
  • pertensor : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended.
  • pergroup : The non-transposed weight input is recommended.
  • perchannel : When the is ND, the transposed [object Object] input is recommended. When the is FRACTAL_NZ, the non-transposed [object Object] input is recommended. If the value range of m is [65, 96], antiquantScale of the UINT64 or INT64 data type is recommended.

[object Object]

The following example is for reference only. For details, see .