[object Object][object Object][object Object]undefined[object Object]

• API function: [object Object] extends the functionality of [object Object]. Building upon the support for FLOAT16/BFLOAT16 input types for x1 and x2:

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
    • Support for the low-precision data type INT8 is added. It supports per-token/per-channel .

• Formula:

  • Scenario 1: If the data type of x1 and x2 is FLOAT16/BFLOAT16, the input x1 undergoes an AllGather operation, followed by a MatMul operation on x1 and x2.
  $$output=allgather(x1)@x2 + bias$$ $$gatherOut=allgather(x1)$$
  • Scenario 2: If the data type of x1 and x2 is FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8 in the per-tensor scenario, or if the data type of x1 and x2 is INT8 in the per-channel or per-token scenario, and amaxOut is not output, the input x1 undergoes an AllGather operation, followed by a MatMul operation on x1 and x2, and finally a dequantization operation.
  $$output=(x1Scale*x2Scale)*(allgather(x1)@x2 + bias)$$ $$gatherOut=allgather(x1)$$

  • Scenario 3: If groupSize is set to a valid value, the input x1 undergoes an AllGather operation, followed by a per-block quantized MatMul operation on x1 and x2, and finally a dequantization operation.

    $$\begin{align*} & output[r(i), r(j)] = \sum_{k=1}^{\frac{K}{groupSizeK}} x1Scale[i, k] * x2Scale[k, j] * (allgather(x1)[r(i), r(j)] @ x2[r(k), r(j)]) \\ & r(z) = (groupSizeK * (z - 1) + 1) : (groupSizeK * z) \\ & output = \begin{bmatrix} output[r(1), r(1)] & \cdots & output[r(1), r(\frac{N}{groupSizeN})] \\ \vdots & \ddots & \vdots \\ output[r(\frac{M}{groupSizeM}), r(1)] & \vdots & output[r(\frac{M}{groupSizeM}), r(\frac{N}{groupSizeN})] \end{bmatrix} \end{align*}$$

    Where $output\left[r(y), r(z)\right]$ represents the block formed by extracting rows $(groupSizeM*(y-1)+1)$ through $(groupSizeM*y)$ and columns $(groupSizeN*(z-1)+1)$ through $(groupSizeN*z)$ from the output matrix.

  • Scenario 4: If the data type of x1 and x2 is FLOAT8_E4M3FN/FLOAT8_E5M2, where the x1 shape is (a0, a1, 2), the x2 shape is (b1, b0, 2), the x1Scale shape is (a0, ceilDiv(a1, 64), 2), and the x2Scale shape is (b1, ceilDiv(b0, 64), 2); assuming x1 is not transposed, x2 is transposed, x1Scale is not transposed, and x2Scale is transposed, the input x1 undergoes an AllGather operation, followed by a MatMul operation on x1 and x2, and finally a dequantization operation.

    $$gatherOut=append(allgather(x1), allgather(x1Scale))$$ $$output=\sum_{0}^{\left \lfloor \frac{k}{blockSize=32} \right \rfloor} (allgather(x1)_{pr}@x2_{rq}*(allgather(x1Scale)_{pr}*x2Scale_{rq}))$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  • [object Object] (aclTensor*, compute input): two-dimensional aclTensor on the device, representing the left matrix for matrix multiplication (MM), corresponding to x1 in the formula.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When [object Object] is [object Object], supported data types include FLOAT16 and BFLOAT16. When [object Object] is [object Object], supported data types include FLOAT16, BFLOAT16, and INT8. The is ND. The current version only supports two-dimensional input shapes and non-transposed scenarios.
  • [object Object] (aclTensor*, compute input): two-dimensional aclTensor on the device, representing the right matrix for MM, corresponding to x2 in the formula.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When [object Object] is [object Object], supported data types include FLOAT16 and BFLOAT16. When [object Object] is [object Object], supported data types include FLOAT16, BFLOAT16, and INT8. The shape is [k, n]. The is ND. can be constructed via transposition. The current version only supports two-dimensional input shapes.
  • [object Object] (aclTensor*, compute input): one-dimensional aclTensor on the device, corresponding to bias in the formula.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When [object Object] is [object Object], supported data types include FLOAT16 and BFLOAT16, and the is ND. The current version supports only 0 as input. When [object Object] is [object Object], the current version only supports [object Object] as input.
  • [object Object] (aclTensor*, compute input): aclTensor on the device, representing the dequantization parameter for the left MM matrix.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When [object Object] is [object Object], only [object Object] is supported. When [object Object] is [object Object], the data type is FLOAT, and the is ND. When the data type of x1 and x2 is FLOAT16/BFLOAT16, only [object Object] is supported. In per-token scenarios, the shape is (m, 1).
  • [object Object] (aclTensor*, compute input): aclTensor on the device, representing the dequantization parameter for the right MM matrix.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When [object Object] is [object Object], only [object Object] is supported. When [object Object] is [object Object], the supported data types are FLOAT and INT64, and the is ND. The INT64 data type is only supported when the output data type is FLOAT16. When the data type of x1 and x2 is FLOAT16/BFLOAT16, only [object Object] is supported. In per-channel scenarios, the shape is (1, n).
  • [object Object] (aclTensor*, compute input): one-dimensional aclTensor on the device, representing the quantization parameter for the MM output matrix. The shape is [1], and the data type is FLOAT. The data format is ND. The current version only supports [object Object] as input.
  • [object Object] (int64_t, compute input): integer on the host used to indicate the number of elements along the M and N axes of the MM output matrix that corresponds to a quantization parameter in that direction. [object Object] is composed of blockSizeM, blockSizeN, and blockSizeK. Each value occupies 16 bits, and the formula is [object Object]. The MM output matrix does not involve the K axis, therefore, blockSizeK is fixed at 0. The current version supports only 0 for blockSizeM and blockSizeN.
  • [object Object] (char*, compute input): string on the host identifying the column group, representing the communication domain name. The data type is string. It is obtained via the Hccl interface: [object Object], where [object Object] is the group.
  • [object Object] (int64_t, compute input): integer on the host identifying the gather target: 0 for the left matrix, 1 for the right matrix. The data type is INT64. The current version supports only 0 as input.
  • [object Object] (int64_t, compute input): integer on the host representing the number of splits communication data copies, equal to the value of total data volume divided by single communication volume. The data type is INT64. The current version supports only 0 as input.
  • [object Object] (int64_t, compute input): integer on the host representing the enumeration of stream modes. The data type is INT64. The current version supports only 1 as input.
  • [object Object] (int64_t, Compute input): number of elements in the [object Object] or [object Object] input along a specific dimension that a single value in [object Object] or [object Object] applies to during dequantization. [object Object] is composed of groupSizeM, groupSizeN, and groupSizeK. Each value occupies 16 bits, and the formula is [object Object].
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The current version supports only 0 as input.
  • [object Object] (char*, compute input): char on the host representing the communication mode. The data type is string.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Currently, only the [object Object] mode is supported. In the [object Object] mode, the AI Vector Core is used to complete communication tasks. The current version supports only [object Object] as input.
  • [object Object] (aclTensor*, compute output): aclTensor on the device, representing the result of the all_gather + MM computation, corresponding to output in the formula.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Supported data types include FLOAT16 and BFLOAT16, and the is ND. If the [object Object] data type is FLOAT16 or BFLOAT16, the [object Object] data type is the same as that of [object Object].
  • [object Object] (aclTensor*, compute output): aclTensor on the device that outputs only the result of all_gather communication, corresponding to gatherOut in the formula.
    • For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Supported data types include FLOAT16, BFLOAT16, and INT8, and the data type is the same as that of [object Object] The is ND.
  • [object Object] (aclTensor*, compute output): one-dimensional aclTensor on the device, representing the maximum value result of the MM computation, corresponding to amaxOut in the formula. The shape is [1], and the data type is FLOAT. The current version only supports [object Object] or an empty tensor.
  • [object Object] (uint64_t*, output): integer on the device representing the size of the workspace required to be allocated on the device.

• [object Object] (aclOpExecutor**, output): aclOpExecutor on the device that returns the operator executor, containing the operator computation process.

• Returns

  aclnnStatus status code. For details, see .

  [object Object]

[object Object]

• Parameters

  • [object Object] (void*, input): address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling [object Object].
  • [object Object] (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input): stream for executing the task.

• Returns

  aclnnStatus status code. For details, see .

[object Object]

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
  • The [object Object] matrix cannot be transposed. The [object Object] matrix can be transposed or not transposed.
  • Input [object Object] is 2D (m, k).
  • Input [object Object] must be 2D (k, n). The axes must meet the input parameter requirements of the MatMul operator. The k axes of [object Object] and [object Object] must be equal and fall within the range of [256, 65535).
  • Input [object Object] must be 1D (n,).
  • The output is 2D (m*rank_size, n). rank_size indicates the number of ranks.
  • Empty tensors are not supported.
  • The data types of [object Object] and [object Object] must be the same.
  • Two, four, and eight ranks are supported.

[object Object]

The following example is for reference only. For details, see .