[object Object]

Note: This API will be deprecated in later versions. Use the latest aclnnQuantMatmulV5 API instead.

[object Object][object Object]undefined[object Object]

• Description: Supports K-C && K-T based on the compatibility with aclnnQuantMatmulV3. Performs quantized matrix multiplication, supporting at least two-dimensional input and at most six-dimensional-dimensional input. Similar APIs include aclnnMm (only two-dimensional tensors can be used as the input of matrix multiplication) and aclnnBatchMatMul (only three-dimensional matrix multiplication is supported, whose first dimension is the batch dimension).
• Formula:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950PR/Ascend 950DT:

    • No pertoken, no bias:

      $$out = x1@x2 * scale + offset$$

    • bias (INT32):

      $$out = (x1@x2 + bias) * scale + offset$$

    • bias (BFLOAT16/FLOAT32) (no offset in this scenario):

      $$out = x1@x2 * scale + bias$$

    • With pertoken, no bias:

      $$out = x1@x2 * scale * pertokenScaleOptional$$

    • With pertoken, bias INT32 (no offset in this scenario):

      $$out = (x1@x2 + bias) * scale * pertokenScaleOptional$$

    • With pertoken, bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):

      $$out = x1@x2 * scale * pertokenScaleOptional + bias$$

  • [object Object]Atlas inference products[object Object]:

    • No pertokenScaleOptional and no bias:

      $$out = x1@x2 * scale + offset$$

    • No pertokenScaleOptional and bias INT32:

      $$out = (x1@x2 + bias) * scale + offset$$

    • With pertokenScaleOptional and no bias:

      $$out = x1@x2 * scale * pertokenScaleOptional$$

    • With pertokenScaleOptional and bias INT32:

      $$out = (x1@x2 + bias) * scale * pertokenScaleOptional$$

[object Object]

Each operator has calls. First, aclnnQuantMatmulV4GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnQuantMatmulV4 is called to perform computation.

[object Object]

• Parameters:

  [object Object]

  • [object Object]Atlas inference products[object Object]:

    • The size of the last dimension of x1 or x2 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.
    • The data type of x1 can be INT8.
    • The data type of x2 supports INT8. When x2 is in NZ format, transposeX2 cannot be false. If pertokenScaleOptional is not an empty tensor, aclnnTransMatmulWeight must be called to process x2 whose format is ND to obtain the affinity data layout format of the AI processor.
    • The data type of bias can be INT32.
    • When pertokenScaleOptional is not an empty tensor, the scale data type supports FLOAT32. When pertokenScaleOptional is an empty tensor, the scale data type supports UINT64 and INT64.
    • The output data type supports FLOAT16 and INT8. When pertokenScaleOptional is not an empty tensor, the output data type supports only FLOAT16.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: - The size of the last dimension of x1 or x2 cannot exceed 65535. - The data type of x1 can be INT8, INT32, or INT4. When the data type is INT32 or INT4, INT4 quantization is used. Currently, the only supported scenario is two-dimensional to six-dimensional ND format with transposeX1 = false. When the data type of x1 is INT4, the shape is represented by (batch, m, k), where k must be an even number. When the data type of x1 is INT32, each INT32 data entry stores eight INT4 data entries, with shape represented by (batch, m, k // 8), where k must be a multiple of 8. - The data type of x2 can be INT8, INT32, or INT4. When the data type is INT32 or INT4, the INT4 quantization scenario is used. Currently, only the two-dimensional ND format is supported. - When the data type is INT4, if transposeX2 is true, the shape is represented by (n, k), where k must be an even number; if transposeX2 is false, the shape is represented by (k, n), where n must be an even number. - When the data type is INT32, each INT32 data entry stores eight INT4 data entries. If transposeX2 is true, the shape is represented by (n, k // 8), where k must be a multiple of 8. If transposeX2 is false, the shape is represented by (k, n // 8), where n must be a multiple of 8. - The aclnnConvertWeightToINT4Pack API can be used to convert x2 from INT32 (one int32 space stores one int4 data entry in bits 0–3) to INT32 (one int32 space stores eight int4 data entries) or INT4 (one int4 space stores one int4 data entry). For details, see . - The data type of bias can be INT32, BFLOAT16, FLOAT16, or FLOAT32. When x1 and x2 are INT32 or INT4, the shape of bias can only be one-dimensional (n,). - When x1 and x2 are INT32 or INT4, transposeX1 can only be false. - The data type of out can be FLOAT16, INT8, BFLOAT16, or INT32.

  • Ascend 950PR/Ascend 950DT: - The data type of x1 can be INT8. - The data type of x2 can be INT8. When one of the last two axes is 1 (that is, n = 1 or k = 1), x2 does not support the private format and supports only the ND format. - The data type of bias can be INT32, BFLOAT16, FLOAT16, or FLOAT32. - The data type of out can be FLOAT16, INT8, BFLOAT16, or INT32. - When x2 is in ND format, if the input x1 is an empty tensor with m = 0 or the input x2 is an empty tensor with n = 0, the output is an empty tensor. When x2 is in FRACTAL_NZ format, if the input x1 is an empty tensor with m = 0, the output is an empty tensor.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic description:

  • [object Object]Atlas training products[object Object] and [object Object]Atlas inference products[object Object]: aclnnQuantMatmulV4 defaults to a deterministic implementation.
  • Ascend 950PR/Ascend 950DT: The aclnnQuantMatmulV4 API is implemented in deterministic mode by default.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Before calling this API, you can use the API to process x2 with the format of ND to obtain the data layout format that is affinity to the AI processor.

• Ascend 950PR/Ascend 950DT:

  1. Before this API is called, the x2 tensor with format ND can be processed to obtain the NZ format through or .
  2. If the last two dimensions of the original ND tensor are 1, you are not advised to convert the format to NZ. By default, x2 is discontinuous, and only discontinuous x2 tensors are supported.

The input and output support the following data type combinations:

• [object Object]Atlas inference products[object Object]:

  [object Object]undefined

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  [object Object]undefined

• Ascend 950PR/Ascend 950DT:

  When pertokenScaleOptional is null, T-C && T-T is supported. When pertokenScaleOptional is not null, K-C quantization and K-T are supported.

  [object Object]undefined

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950 PR/Ascend 950 DT:

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: x2 is the NZ format (transposeX2 = false).

  [object Object]

x2 is in NZ format (transposeX2=true).

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: INT4 quantization scenario (x1 and x2 are of the INT4 type, and transposeX2 is false).

  [object Object]