[object Object]

[object Object][object Object]undefined

Note: When using this API, ensure that the driver firmware package and CANN package are in the 8.0.RC2 version or later. Otherwise, an error, such as BUS ERROR, will be reported.

[object Object]

• API function: Extends the functions of the [object Object] API and supports low-bit communication: the MatMul computation results undergo AlltoAll communication, ReduceSum computation, AllGather communication, and dequantization in sequence, replacing the original workflow of dequantization and per-token scaling followed by AllReduce communication. It supports per-tensor, per-channel, and per-token .

• Formula:

  This API handles the following scenarios:

  • Case 1: Perform MatMul computation on the quantized inputs x1 and x2, then perform dequantization, then perform Add operation with x3, and finally perform AllReduce computation.

    $$output= AllReduce(dequantScale*(x1_{int8}@x2_{int8} + bias_{int32}) + x3)$$

  • Case 2: Perform MatMul computation on the quantized inputs x1 and x2, then perform dequantization and pertoken computation, then perform Add operation with x3, and finally perform AllReduce computation.

    $$output= AllReduce(dequantScale * pertokenScaleOptional * (x1_{int8}@x2_{int8} + biasOptional_{int32}) + x3Optional)$$

  • Case 3: Perform MatMul, dequant, and pertoken calculation on the quantized input parameters x1 and x2, perform the Add operation on x3, perform perchannel quantization on the output, perform AllToAll communication, and perform ReduceSum calculation on the first communication result, then, AllGather communication is performed. Finally, dequantization is performed on the second communication result to obtain the final output.

    $$matmulAddOutPut = (dequantScale * pertokenScaleOptional * (x1_{int8}@x2_{int8} + biasOptional_{int32}) + x3Optional);$$ $$alltoallOutPut_{int8} = AllToAll(matmulAddOutPut / commQuantScale1Optional);$$ $$reduceSumOutPut_{int8} = (add(alltoallOutPut_{int8}) * (commQuantScale1Optional / commQuantScale2Optional));$$ $$outPut = (AllGather(reduceSumOutPut_{int8}) * commQuantScale2Optional);$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • The input x2 supports the ND and FRACTAL_NZ formats. The rules for the input shape are as follows:
      • When the format of [object Object] is FRACTAL_NZ, the current version supports only 4D input. [object Object] and [object Object] are used to convert the input data format from ND into NZ. Non-contiguous tensors support only the transpose scenario.
      • When the format of [object Object] is ND, the current version supports only 2D input.

  • Ascend 950PR/Ascend 950DT:

    • The data format of x2 can only be ND. (The current version supports only 2D input.)

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: aclnnQuantMatmulAllReduceV3 is implemented in non-deterministic mode by default. You can enable deterministic computing by setting the HCCL_DETERMINISTIC environment variable to true.
  • Ascend 950PR/Ascend 950DT: [object Object] defaults to a deterministic implementation.
• MC2 is disabled in incremental generation scenarios but enabled in full generation scenarios.
• [object Object] can be 2D (m, k) or 3D (b, s, k) and cannot be empty. [object Object] must be 2D and cannot be empty. (k, n), where the k axis meets the input parameter requirements of the MatMul operator. The k axes of [object Object] and [object Object] must be equal.
• The value of m cannot exceed 2147483647. The size of the last dimension of [object Object] (k) and [object Object] (k when transposed and n when not transposed) cannot exceed 65535.
• The passed [object Object], [object Object], [object Object], or [object Object] cannot be a null pointer.
• The data types and formats of [object Object], [object Object], [object Object], [object Object], [object Object] (when not empty), and [object Object] (when not empty) must be supported.
• If [object Object] is of FLOAT16 type, the type of [object Object] is INT64 or UINT64 if [object Object] is empty, or FLOAT32 if [object Object] is not empty. If [object Object] is of BFLOAT16 type, the types of [object Object] and [object Object] are both BFLOAT16.
• If the passed [object Object] and [object Object] are not null pointers, their shapes must be the same, their types must be the same as the operator output type, and the inputs for each rank must be the same.
• If the shape of [object Object] is (b, s, k), the shape of [object Object] is (b × s). If the shape of [object Object] is (m, k), the shape of [object Object] is (m).
• Only the all-mesh networking of HCCS links is supported.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: 1, 2, 4, or 8 ranks are supported.
  • Ascend 950PR/Ascend 950DT: 1, 2, 4, 8, 16, 32, or 64 cards are supported.
• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: supports only one communication domain for MC2 operators within a model.
• INT8 low-bit communication brings performance benefits only when the system is communication-bounded. You are not advised to enable INT8 low-bit communication when the system is computing-bounded, that is, do not input [object Object] or [object Object].
• Support for empty tensors:
  • Empty tensors are not supported.

[object Object]

The following example is for reference only. For details, see .

Note: This sample code calls some HCCL collective communication library APIs: HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT:

  [object Object]