[object Object]

[object Object][object Object]undefined[object Object]

• API function: In the micro-batch training scenario, micro-batch gradient accumulation is required, and there are a large number of fusion scenarios where GroupedMatMul is followed by InplaceAdd. The QuantGroupedMatmulInplaceAdd operator is introduced to fuse the preceding operators to improve the network performance. Performs group matrix multiplication and addition. The basic function is the combination of matrix multiplication and addition. For example, in the T-C quantization scenario, $y_i[m,n]=(x1_i[m,k_i] \times x2_i[k_i,n]) * scale2_i[n] * scale1_i + y_i[m,n], i=1...g$, where g indicates the number of groups, and $m/k_i/n$ indicates the corresponding dimension.

  Compared with the API, this API has the following changes:

  • The input and output parameter types are both aclTensor.
  • The InplaceAdd computation is added after the GroupedMatMul computation is complete.
  • Only quantization scenarios (1.MX quantization; 2.T-C quantization) are supported. For details about the quantization modes, see .
  • Only x1 and x2 of FLOAT8_E5M2, FLOAT8_E4M3FN and HIFLOAT8 are supported.

• Formulas:

  • MX quantization:
  $$y_i[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice_i * x2Slice_i)) * (scale1_i[m, j] * scale2_i[j, n])) + y_i[m,n]$$

  In the preceding information, gsK indicates the quantized block size of the K axis, that is, 32. $x1Slice_i$ indicates the vector whose length is gsK and that is in the mth row of $x1_i$. $x2Slice_i$ indicates the vector whose length is gsK and that is in the nth column of $x2_i$. The K axis is sliced from $j*gsK$. The value range of j is [0, kLoops), and kLoops is calculated as follows: kLoops = ceil($K_i$ / gsK). The length of the last slice can be less than gsK.

  • T-C quantization:
  $$y_i=(x1_i\times x2_i) * scale2_i * scale1_i + y_i$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Determinism description: [object Object] defaults to a deterministic implementation.

• The size of each dimension of x1 and x2 must be less than the maximum value 2147483647 of int32 after 32-byte alignment, and the size of the inner axis must be less than 2097152.

  • The supported input types in the dynamic quantization (T-C quantization) scenario are as follows:

    • The data type combinations supported by non-empty parameters must meet the requirements listed in the following table.

      [object Object]undefined

    • The scale1Optional/scale2 must meet the following constraints (g indicates the number of matmul groups, that is, the number of groups):

      [object Object]undefined

  • The supported data types in the dynamic quantization (mx quantization) scenario are as follows:

    • The data type combinations must meet the requirements listed in the following table.

      [object Object]undefined

    • The scale1Optional/scale2 must meet the following constraints (g indicates the number of matmul groups, that is, the number of groups, and g_i indicates the ith group (the subscript starts from 0)):

      [object Object]undefined

• The maximum value of the first dimension of groupList is 1024, that is, a maximum of 1024 groups are supported.

[object Object]

The following example is for reference only. For details, see .