[object Object]

[object Object][object Object]undefined[object Object]

• API usage: In the micro-batch training scenario, gradients need to be accumulated in the micro-batch. As a result, there are a large number of fusion scenarios where QuantBatchMatmul is followed by InplaceAdd. The QuantBatchMatmulInplaceAdd operator is used to fuse the preceding operators to improve the network performance. Performs quantized matrix multiplication and addition. The basic function is the combination of matrix multiplication and addition.

• Formula:

  • mx quantization:
  $$y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n]$$

  $gsK$ indicates the quantized block size of the K axis, that is, 32. $x1Slice$ indicates the vector of length $gsK$ in row m of $x1$. $x2Slice$ indicates the vector of length $gsK$ in column n of $x2$. The K axis is sliced from $j*gsK$. The value range of j is [0, kLoops), and kLoops = ceil($K_i$/$gsK$). The length of the last slice can be less than $gsK$.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • Formula: [object Object][object Object]$$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$$

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic description: The default deterministic implementation of aclnnQuantBatchMatmulInplaceAdd is used.
• Currently, only transposeX1 is true and transposeX2 is false.
• Restrictions on groupSize:
  • The input groupSize is decomposed into groupSizeM, groupSizeN, and groupSizeK according to the following formulas. If one or more of them are 0, groupSizeM, groupSizeN, and groupSizeK are reset based on the input shape of x1/x2/x1Scale/x2Scale for computation. Principle: If groupSizeM is 0, the quantization group value in the m direction is inferred by the API. The inference formula is groupSizeM = m/scaleM (m must be exactly divided by scaleM). m is the same as that in the shape of x1, and scaleM is the same as that in the shape of x1Scale.$$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$$
• Restrictions on dynamic quantization (mx quantization):

  • The following table describes the supported input and output data type combinations.

    [object Object]undefined

  • The value relationships between x1 data type, x2 data type, x1, x2, x1Scale, x2Scale, and groupSize are as follows:

    [object Object]undefined

[object Object]

The following example is for reference only. For details, see .