[object Object][object Object][object Object]undefined[object Object]

• Function: fuses the quantized matrix multiplication and GELU computation.

• Formula:

  • With x1Scale, bias (INT32) (no offset in this scenario):

    $$qbmmout = (x1@x2 + bias) * x2Scale * x1Scale$$

  • With x1Scale, bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):

    $$qbmmout = x1@x2 * x2scale * x1Scale + bias$$

  • With x1Scale, no bias:

    $$qbmmout = x1@x2 * x2Scale * x1Scale$$

  • The operator type is defined by the input fusedOpType. The following types are supported:

    • gelu_tanh operation:

      $$out = gelu\_tanh(qbmmout)$$

    • gelu_erf operation:

      $$out = gelu\_erf(qbmmout)$$

[object Object]

Each operator has calls. You must call aclnnFusedQuantMatmulWeightNzGetWorkspaceSize to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call aclnnFusedQuantMatmulWeightNz to perform the computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic description:

  • For [object Object]Atlas training products[object Object] and [object Object]Atlas inference products[object Object], the default deterministic implementation of aclnnFusedQuantMatmulWeightNz is used.

• The following table describes the supported input and output data type combinations.

  [object Object]undefined

• Currently, the API supports x1 per-token quantization and x2 per-channel/per-tensor quantization. The input dtype combinations of x1, x2, x1Scale, and x2Scale supported by different quantization modes are as follows:

  • The data type of x1 can be INT8, INT32, or INT4.
    • When the data type is INT32 or INT4, the INT4 quantization scenario is used.
      • Currently, only the ND format is supported.
      • Currently, only non-transposed inputs are supported.
      • The inner axis of x1 must be an even number.
    • When the data type is INT32, each INT32 data entry stores eight INT4 data entries, with shape (batch, m, k // 8), where k must be a multiple of 8.
  • The data type of x2 can be INT8, INT32, or INT4.
    • This API supports only the x2 data in NZ format. In this case, k and n cannot be 1.
    • When the data type is INT32, eight INT4 data elements are stored in each INT32 data element.
    • The aclnnConvertWeightToINT4Pack API can be used to convert x2 from INT32 (one int32 space stores one int4 data entry in bits 0–3) to INT32 (one int32 space stores eight int4 data entries) or INT4 (one int4 space stores one int4 data entry). For details, see .
    • In AI processor affinity data layout format, the shape can be four- to eight-dimensional.
      • The dimension for transposition is (batch, k1, n1, n0, k0), where batch may not exist, k0 = 32, and n0 = 16. The k in the x1 shape and k1 in the x2 shape must meet the following relationship: ceil(k/32) = k1.
      • The dimension for non-transposition is (batch, n1, k1, k0, n0), where batch may not exist, k0 = 16, and n0 = 32. The k in the x1 shape and k1 in the x2 shape must meet the following relationship: ceil(k/16) = k1.
      • aclnnCalculateMatmulWeightSizeV2 and aclnnTransMatmulWeight can be used to convert the input from ND format to AI processor affinity data layout format.
  • The constraints for x1Scale are as follows:
    • The shape supports one dimension, and the shape is (m,). The data type can be FLOAT32.
  • The constraints for x2Scale are as follows:
    • The shape supports one dimension, which is (n,) or (1,). The value of n is the same as that of x2. The supported data types are FLOAT32 and BFLOAT16.
  • The constraints on biasOptional are as follows:
    • The shape supports one or three dimensions. In INT4 quantization scenarios, biasOptional supports only one dimension, and the shape is (n). When the shape is three-dimensional, the shape of biasOptional is (batch, 1, n).
    • The supported data types are int32, float32, bfloat16, and float16.
  • The constraints for out are as follows:
    • The shape supports 2 to 6 dimensions, that is, (batch, m, n). The data type can be FLOAT16 or BFLOAT16.

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: x1 is of type INT8, x2 is of type INT8, x1Scale is of type FLOAT32, and x2Scale is of type FLOAT32.

  [object Object]