[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs matrix multiplication for quantization. It is compatible with the aclnnQuantMatmulV3 and aclnnQuantMatmulV4 APIs. It supports at least one-dimensional input and at most two-dimensional input. Similar APIs include aclnnMm (only two-dimensional tensors can be used as the input of matrix multiplication).
• Formula:

  • x1 is INT8, x2 is INT32, x1Scale is FLOAT32, x2Scale is UINT64, and yOffset is FLOAT32:

    $$out = ((x1 @ (x2*x2Scale)) + yOffset) * x1Scale$$

  • No x1Scale, no bias:

    $$out = x1@x2 * x2Scale + x2Offset$$

  • bias (INT32):

    $$out = (x1@x2 + bias) * x2Scale + x2Offset$$

  • bias (BFLOAT16/FLOAT32) (no offset in this scenario):

    $$out = x1@x2 * x2Scale + bias$$

  • With x1Scale, no bias:

    $$out = x1@x2 * x2Scale * x1Scale$$

  • With x1Scale, bias (INT32) (no offset in this scenario):

    $$out = (x1@x2 + bias) * x2Scale * x1Scale$$

  • With x1Scale, bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):

    $$out = x1@x2 * x2Scale * x1Scale + bias$$

  • x1 and x2 are INT8, x1Scale and x2Scale are FLOAT32, bias is FLOAT32, and out is FLOAT16 or BFLOAT16 (pergroup-perblock quantization):

    $$out = (x1 @ x2) * x1Scale * x2Scale + bias$$

  • x1 and x2 are INT4, x1Scale and x2Scale are FLOAT32, x2Offset is FLOAT16, and out is FLOAT16 or BFLOAT16 (pertoken-pergroup asymmetric quantization):

    $$out = x1Scale * x2Scale @ (x1 @ x2 - x1 @ x2Offset)$$

[object Object]

Each operator has calls. First, aclnnQuantMatmulV5GetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnQuantMatmulV5 is called to perform computation.

[object Object]

• Parameters:

  [object Object]

  • Formula 1:

    $$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$$

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]:

    The size of the last dimension of x1 or x2 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.

• Returns:

  aclnnStatus: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  aclnnStatus: status code. For details, see .

[object Object]

• Determinism description: aclnnQuantMatmulV5 defaults to a deterministic implementation.

• The input and output support the following data type combinations:

  [object Object]undefined

• The input dtype combinations of x1, x2, x1Scale, and x2Scale and the supported platforms in different quantization modes are as follows:

  The dtype and shape values of x1, x2, x1Scale, x2Scale, yOffset, and groupSize affect each other in different quantization scenarios. The relationships are as follows:

  [object Object]undefined

• The input combinations that belong to APIs aclnnQuantMatmulV3 and aclnnQuantMatmulV4 are not listed in the table. These two APIs do not support the input groupsize, and the default value of groupsize is 0.

  • The constraints for x1 are as follows:

    • The data type can be INT8, INT32, or INT4.
    • When the data type is INT32 or INT4, the INT4 quantization scenario is used, and transposeX1 is set to false.
    • When the data type is INT4, the shape is (batch, m, k), where k must be an even number.
    • When the data type is INT32, each INT32 data entry stores eight INT4 data entries, with shape (batch, m, k // 8), where k must be a multiple of 8.
    • When A8W8 perblock symmetric quantization is used, the data type can be INT8. Currently, n must be a multiple of 256, k must be a multiple of both 128 and 4 × 128, transposeX1 is set to false, and the shape is (m, k).
    • When A4W4 pergroup asymmetric quantization is used, the data type can be INT4. Currently, k must be a multiple of 1024, transposeX1 is set to false, and the shape is (m, k).
    • When transposeX1 is set to false, the shape is (batch, m, k).
    • When transposeX1 is set to true, the shape is (batch, k, m), where batch represents the first one to four dimensions, and dimension 0 indicates that batch does not exist.
    • In AI processor affinity data layout format, the shape can be four- to eight-dimensional.
    • When transposeX2 is true, the shape is (batch, k1, n1, n0, k0), where batch is optional, k0 = 32, and n0 = 16. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil (k/32) = k1.
    • When transposeX2 is false, the shape is (batch, n1, k1, k0, n0), where batch is optional, k0 = 16, and n0 = 32. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil(k/16) = k1.
    • aclnnCalculateMatmulWeightSizeV2 and aclnnTransMatmulWeight can be used to convert the input from ND format to AI processor affinity data layout format.

  • The constraints for x2 are as follows:

    • The data type can be INT8, INT32, or INT4.
    • When the data type is INT32 or INT4, the INT4 quantization scenario is used. Currently, only the two-dimensional ND format is supported. In ND format, the shape can be two- to six-dimensional.
      • When transposeX2 is set to false, the shape is (batch, k, n).
      • When transposeX2 is set to true, the shape is (batch, n, k).
      • batch is optional and k is the same as that in shape of x1.
    • When the data type is INT4:
      • If transposeX2 is set to true, the shape is (n, k), where k must be an even number.
      • If transposeX2 is set to false, the shape is (k, n), where n must be an even number.
    • When the data type is INT32, each INT32 data entry stores eight INT4 data entries.
      • If transposeX2 is set to true, the shape is (n, k // 8), where k must be a multiple of 8.
      • If transposeX2 is set to false, the shape is (k, n // 8), where n must be a multiple of 8.
    • The aclnnConvertWeightToINT4Pack API can be used to convert x2 from INT32 (one int32 space stores one int4 data entry in bits 0–3) to INT32 (one int32 space stores eight int4 data entries) or INT4 (one int4 space stores one int4 data entry). For details, see .
    • When A8W8 perblock symmetric quantization is used, the data type can be INT8, transposeX2 is set to true, and the shape is (n, k). Currently, n must be a multiple of 256, and k must be a multiple of both 128 and 4 × 128.
    • When A4W4 pergroup asymmetric quantization is used, the data type can be INT4, transposeX2 is set to true, and the shape is (n, k). Currently, k must be symmetric with 1024, and n must be a multiple of 256.

  • The constraints for x1Scale are as follows:

    • The shape is two-dimensional and is represented by (m, 1). The data type can be FLOAT32.
    • When A8W8 perblock symmetric quantization is used, the data type is FLOAT32 and the shape is (m, ceil(k/128)).
    • When A4W4 pergroup asymmetric quantization is used, the data type is FLOAT32 and the shape is (m, 1).

  • The constraints for x2Scale are as follows:

    • The shape is two-dimensional and is represented by (k / groupSize, n), where n is the same as that of x2.
    • The data type can be UINT64, INT64, FLOAT32, or BFLOAT16.
    • TransQuantParamV2 can only be one-dimensional. Therefore, if the original input type does not conform with the data type combinations described in , you need to convert x2_scale view to one-dimensional (k / groupSize * n), call the aclnn API of the TransQuantParamV2 operator to convert x2Scale to the UINT64 data type, and then convert the output view to two-dimensional (k / groupSize, n).
    • When A8W8 perblock symmetric quantization is used, the data type is FLOAT32. If transpose of x2 is true, the shape is (ceil(n/128), ceil(k/128)). If transpose of x2 is false, the shape is (ceil(k/128), ceil(n/128)).
    • When A4W4 pergroup asymmetric quantization is used, the data type is FLOAT32. If transpose of x2 is false, the shape is (ceil(k / 256), n).

  • yScale is not supported in the current version. Pass nullptr.

  • The constraints for x2Offset are as follows:

    • Optional quantization parameter, the data type can be FLOAT32.
    • If the output data type is INT8, the offset can exist. For other input types, nullptr needs to be passed.
    • When A4W4 pergroup asymmetric quantization is used, the input type can be FLOAT16 and the shape is (ceil(k, 256), n).

  • The constraints for bias are as follows:

    • When A8W8 perblock symmetric quantization is used, the data type can be FLOAT32 and the shape can be one-dimensional (n,).
    • A4W4 pergroup is not supported in the current version. Pass nullptr.

  • transposeX1: When x1 and x2 are INT32 or INT4, transposeX1 can only be false and the shape is represented by (m, k).

  • The constraints for transposeX2 are as follows:

    • In ND format, if this parameter is false, the shape is (batch, k, n); if this parameter is true, the shape is (batch, n, k), where batch is optional and k is the same as that in the shape of x1.
    • In AI processor affinity data layout format:
      • If this parameter is true, the shape is (batch, k1, n1, n0, k0), where batch is optional, k0 = 32, and n0 = 16. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil (k/32) = k1.
      • If this parameter is false, the shape is (batch, n1, k1, k0, n0), where batch is optional, k0 = 16, and n0 = 32. k in the shape of x1 and k1 in the shape of x2 must meet the following relationship: ceil(k/16) = k1.

  • The constraints for groupSize are as follows:

    • In common cases and A4W4 pergroup asymmetric quantization mode, the value combination of [groupSizeM, groupSizeN, groupSizeK] can only be [0, 0, 256]. That is, groupSizeK can only be 256.
    • In A8W8 perblock symmetric quantization mode, the value combination of [groupSizeM, groupSizeN, groupSizeK] can only be [1, 128, 128].

  • The constraints for out are as follows:

    • The shape is two-dimensional and is represented by (m, n). The data type can be FLOAT16, INT8, BFLOAT16, or INT32.
    • In A8W8 perblock symmetric quantization mode, the output supports BFLOAT16.
    • In A4W4 pergroup asymmetric quantization mode, the output supports BFLOAT16.

[object Object]

The following example is for reference only. For details, see .

x1 and x2 are FLOAT8_E4M3FN, x1Scale and x2Scale are FLOAT32, x2Offset is not available, and bias is FLOAT32.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object]: x1 is INT8, x2 is INT32, x1Scale is FLOAT32, and x2Scale is UINT64

  [object Object]