[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs quantized matrix multiplication, supporting at least two-dimensional input and at most six-dimensional input. Similar APIs include aclnnMm (only two-dimensional tensors can be used as the input of matrix multiplication) and aclnnBatchMatMul (only three-dimensional matrix multiplication is supported, whose first dimension is the batch dimension). It supports the T-C && T-T .
• Formula:

  • No bias:

    $$out = x1@x2 * scale + offset$$

  • bias (INT32):

    $$out = (x1@x2 + bias) * scale + offset$$

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: bias BFLOAT16/FLOAT32 is supported (no offset in this scenario).

    $$out = x1@x2 * scale + bias$$

[object Object]

Each operator has calls. First, aclnnQuantMatmulV3GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnQuantMatmulV3 is called to perform computation.

• [object Object]

• [object Object]

[object Object]

• Parameters:

  • x1 (aclTensor*, compute input): input x1 in the formula, aclTensor on the device. The can be ND. are supported only when the last two axes are transposed. The shape can be two- to six-dimensional. When transposeX1 is false, the shape is represented by (batch, m, k). When transposeX1 is true, the shape is represented by (batch, k, m), where batch is optional.

    • [object Object]Atlas inference series products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The size of the last dimension of x1 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false.
    • [object Object]Atlas inference series products[object Object]: The data type can be INT8.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The data type can be INT8, INT32, or INT4. When the data type is INT32 or INT4, the INT4 quantization scenario is used. Currently, transposeX1 can only be set to false. When the data type of x1 is INT4, the shape is represented by (batch, m, k), where k must be an even number. When the data type of x1 is INT32, each INT32 data entry stores eight INT4 data entries, with shape represented by (batch, m, k // 8), where k must be a multiple of 8.

  • x2 (aclTensor*, compute input): input x2 in the formula, aclTensor on the device. The can be ND format or AI processor affinity data layout format. In ND format, non-contiguous tensors are supported only when the last two axes are transposed. In other scenarios, are not supported

    • [object Object]Atlas inference series products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The size of the last dimension of x2 cannot exceed 65535. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.
    • In ND format, the shape can be two- to six-dimensional. When transposeX2 is false, the shape is represented by (batch, k, n). When transposeX2 is true, the shape is represented by (batch, n, k), where batch is optional and k is the same as that in the shape of x1.
    • In AI processor affinity data layout format, the shape can be four- to eight-dimensional. When transposeX2 is true, the shape is represented by (batch, k1, n1, n0, k0), where batch is optional, k0 = 32, and n0 = 16. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil (k/32) = k1. When transposeX2 is false, the shape is represented by (batch, n1, k1, k0, n0), where batch is optional, k0 = 16, and n0 = 32. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil (k/16) = k1. aclnnCalculateMatmulWeightSizeV2 and aclnnTransMatmulWeight can be used to convert the input from ND format to AI processor affinity data layout format.
    • [object Object]Atlas inference series products[object Object]: The data type can be INT8. If the input x2 is in AI processor affinity data layout format, transposeX2 cannot be set to false.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The data type can be INT8, INT32, or INT4. When the data type is INT32 or INT4, the INT4 quantization scenario is used. Currently, only the two-dimensional ND format is supported.
      • When the data type is INT4, if transposeX2 is true, the shape is represented by (n, k), where k must be an even number; if transposeX2 is false, the shape is represented by (k, n), where n must be an even number.
      • When the data type is INT32, each INT32 data entry stores eight INT4 data entries. If transposeX2 is true, the shape is represented by (n, k // 8), where k must be a multiple of 8. If transposeX2 is false, the shape is represented by (k, n // 8), where n must be a multiple of 8.
      • The aclnnConvertWeightToINT4Pack API can be used to convert x2 from INT32 (one int32 space stores one int4 data entry in bits 0–3) to INT32 (one int32 space stores eight int4 data entries) or INT4 (one int4 space stores one int4 data entry). For details, see .

  • scale (aclTensor*, compute input): quantization parameter, scale in the formula, aclTensor on the device. The can be ND. The shape is one-dimensional (t,), and t = 1 or n, where n is the same as n of x2.

    • [object Object]Atlas inference series products[object Object]: The data type can be UINT64 or INT64.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The data type can be UINT64, INT64, FLOAT32, or BFLOAT16.
    • If the original input type does not conform with the data type combinations described in , call the aclnn API of the TransQuantParamV2 operator to convert scale to the INT64 or UINT64 type in advance.

  • offset (aclTensor*, compute input): offset in the formula, aclTensor on the device. The data type can be FLOAT32. The can be ND. The shape is one-dimensional (t,), and t = 1 or n, where n is the same as n of x2. If the output data type is INT8, the offset can exist. For other input types, nullptr needs to be passed.

  • bias (aclTensor*, compute input): bias in the formula, aclTensor on the device. This is an optional parameter. The can be ND. The shape can be one-dimensional (n,) or three-dimensional (batch, 1, n), where n is the same as n of x2. When the shape of out is two-, four-, five-, or six-dimensional, the shape of bias can only be one-dimensional (n,).

    • [object Object]Atlas inference series products[object Object]: The data type can be INT32.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The data type can be INT32, BFLOAT16 or FLOAT32. When x1 and x2 are INT32 or INT4, the shape of bias can only be one-dimensional (n,).

  • transposeX1 (bool, compute input): whether the input shape of x1 is transposed. When transposeX1 is false, the shape is represented by (batch, m, k). When transposeX1 is true, the shape is represented by (batch, k, m), where batch is optional.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: If x1 and x2 are INT32 or INT4, transposeX1 can only be false.

  • transposeX2 (bool, compute input): whether the input shape of x2 is transposed.

    • In ND format, if transposeX2 is false, the shape is represented by (batch, k, n); if transposeX2 is true, the shape is represented by (batch, n, k), where batch is optional and k is the same as that in the shape of x1.
    • In AI processor affinity data layout format, if transposeX2 is true, the shape is represented by (batch, k1, n1, n0, k0), where batch is optional, k0 = 32, and n0 = 16. k in the shape of x1, and k1 in the shape of x2 must meet the following relationship: ceil (k/32) = k1. When transposeX2 is false, the shape is represented by (batch, n1, k1, k0, n0), where batch is optional, k0 = 16, and n0 = 32. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil (k/16) = k1.

  • out (aclTensor*, compute output): out in the formula, aclTensor on the device. The can be ND. are supported. The shape can be two- to six-dimensional, and is represented by (batch, m, n), where batch is optional. The batch dimensions of x1 and x2 can be broadcast, the output batch is the same as the broadcast batch, m is the same as the m of x1, and n is the same as the n of x2.

    • [object Object]Atlas inference series products[object Object]: The data type can be FLOAT16 or INT8.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The data type can be FLOAT16, INT8, BFLOAT16, or INT32.

  • workspaceSize (uint64_t*, output): size of the workspace required to be allocated on the device.

  • executor (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Parameters:

  • workspace (void*, input): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by the first-phase API aclnnQuantMatmulV3GetWorkspaceSize.
  • executor (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • stream (aclrtStream, input): stream for executing the task.

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Deterministic description:

  • [object Object]Atlas training series products[object Object] and [object Object]Atlas inference series products[object Object]: aclnnQuantMatmulV3 defaults to a deterministic implementation.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object], and [object Object]Atlas inference series products[object Object]: Before calling this API, you can use to process x2 in ND format to obtain x2 in AI processor affinity data layout format. The following data type combinations are supported for the input and output. The combinations support T-C && T-T .

• [object Object]Atlas inference series products[object Object]:

  [object Object]undefined

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:

  [object Object]undefined

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The following example is for reference only. For details, see .

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The sample code (transposeX2 = false) when x2 is in AI processor affinity data layout format is as follows (for reference only). For details about the compilation and running process, see .

  [object Object]

• [object Object]Atlas inference series products[object Object]: The sample code (transposeX2 = true) when x2 is in AI processor affinity data layout format is as follows (for reference only). For details about the compilation and running process, see .

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: The sample code (x1 and x2 are int4, transposeX2 = false) in the INT4 quantization scenario is as follows (for reference only). For details about the compilation and running process, see .

  [object Object]