[object Object]

[object Object][object Object]undefined[object Object]

• Description: Supports K-C && K-T based on the compatibility with aclnnQuantMatmulV3. Performs quantized matrix multiplication, supporting at least two-dimensional input and at most six-dimensional-dimensional input. Similar APIs include aclnnMm (only two-dimensional tensors can be used as the input of matrix multiplication) and aclnnBatchMatMul (only three-dimensional matrix multiplication is supported, whose first dimension is the batch dimension).
• Formula:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:

    • No pertoken, no bias:

      $$out = x1@x2 * scale + offset$$

    • bias (INT32):

      $$out = (x1@x2 + bias) * scale + offset$$

    • bias (BFLOAT16/FLOAT32) (no offset in this scenario):

      $$out = x1@x2 * scale + bias$$

    • With pertoken, no bias:

      $$out = x1@x2 * scale * pertokenScaleOptional$$

    • With pertoken, bias INT32 (no offset in this scenario):

      $$out = (x1@x2 + bias) * scale * pertokenScaleOptional$$

    • With pertoken, bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):

      $$out = x1@x2 * scale * pertokenScaleOptional + bias$$

  • [object Object]Atlas inference series products[object Object]:

    • No bias:

      $$out = x1@x2 * scale + offset$$

    • bias int32:

      $$out = (x1@x2 + bias) * scale + offset$$

[object Object]

Each operator has calls. First, aclnnQuantMatmulV4GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnQuantMatmulV4 is called to perform computation.

[object Object]

• Parameters:

  [object Object]
  • [object Object]Atlas inference series products[object Object]:
    • The size of the last dimension of x1 or x2 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.
  • The data type of x1 can be INT8.
    • The data type of x2 can be INT8. When the data format is AI processor affinity, transposeX2 cannot be false.
    • The data type of bias can be INT32.
    • The data type of scale can be UINT64 or INT64.
    • pertokenScaleOptional is not supported.
    • The data type of out can be FLOAT16 or INT8.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:
    • The size of the last dimension of x1 or x2 cannot exceed 65535.

• The data type of x1 can be INT8, INT32, or INT4. When the data type is INT32 or INT4, INT4 quantization is used. Currently, the only supported scenario is two-dimensional to six-dimensional ND format with transposeX1 = false. When the data type of x1 is INT4, the shape is represented by (batch, m, k), where k must be an even number. When the data type of x1 is INT32, each INT32 data entry stores eight INT4 data entries, with shape represented by (batch, m, k // 8), where k must be a multiple of 8.

  • The data type of x2 can be INT8, INT32, or INT4. When the data type is INT32 or INT4, the INT4 quantization scenario is used. Currently, only the two-dimensional ND format is supported.
  • When the data type is INT4, if transposeX2 is true, the shape is represented by (n, k), where k must be an even number; if transposeX2 is false, the shape is represented by (k, n), where n must be an even number.
  • When the data type is INT32, each INT32 data entry stores eight INT4 data entries. If transposeX2 is true, the shape is represented by (n, k // 8), where k must be a multiple of 8. If transposeX2 is false, the shape is represented by (k, n // 8), where n must be a multiple of 8.
  • The aclnnConvertWeightToINT4Pack API can be used to convert x2 from INT32 (one int32 space stores one int4 data entry in bits 0–3) to INT32 (one int32 space stores eight int4 data entries) or INT4 (one int4 space stores one int4 data entry). For details, see .
  • The data type of bias can be INT32, BFLOAT16, FLOAT16, or FLOAT32. When x1 and x2 are INT32 or INT4, the shape of bias can only be one-dimensional (n,).
  • When x1 and x2 are INT32 or INT4, transposeX1 can only be false.
  • The data type of out can be FLOAT16, INT8, BFLOAT16, or INT32.

• The data type of x1 can be INT8.

  • The data type of x2 can be INT8. When one of the last two axes is 1 (that is, n = 1 or k = 1), x2 does not support the private format and supports only the ND format.
  • The data type of bias can be INT32, BFLOAT16, FLOAT16, or FLOAT32.
  • The data type of out can be FLOAT16, INT8, or BFLOAT16.

• Returns:

  aclnnStatus: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Deterministic description:

  • [object Object]Atlas training series products[object Object] and [object Object]Atlas inference series products[object Object]: aclnnQuantMatmulV4 defaults to a deterministic implementation.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object], and [object Object]Atlas inference series products[object Object]: Before calling this API, you can use to process x2 in ND format to obtain x2 in AI processor affinity data layout format. The input and output support the following data type combinations:

• [object Object]Atlas inference series products[object Object]:

  [object Object]undefined

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:

  [object Object]undefined

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: x2 is in AI processor affinity data layout format and transposeX2 is false.

  [object Object]

• [object Object]Atlas inference series products[object Object]: x2 is in the AI processor affinity data layout format and transposeX2 is true.

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: INT4 quantization scenario (x1 and x2 are of the INT4 type, and transposeX2 is false).

  [object Object]