[object Object]

[object Object][object Object]undefined

Note: When using this API, ensure that the driver firmware package and CANN package are in the 8.0.RC2 version or later. Otherwise, an error, such as BUS ERROR, will be reported.

[object Object]

• API function: Extends the functions of the [object Object] API and supports the per-token quantization mode. It supports per-tensor, per-channel, and per-token .

• Formula:

  This API handles the following scenarios:

  • Case 1: Perform MatMul on the quantized inputs x1 and x2, then perform dequantization, then perform Add with x3, and finally perform AllReduce.

    $$output= AllReduce(dequantScale*(x1_{int8}@x2_{int8} + bias_{int32}) + x3)$$

  • Case 2: Perform MatMul on the quantized inputs x1 and x2, then perform dequantization and pertoken quantization, then perform Add with x3, and finally perform AllReduce.

    $$output= AllReduce(dequantScale * pertokenScaleOptional * (x1_{int8}@x2_{int8} + biasOptional_{int32}) + x3Optional)$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • The input x2 supports the ND format (only two-dimensional input is supported in the current version) and FRACTAL_NZ format (only four-dimensional input is supported in the current version). When the format of [object Object] is FRACTAL_NZ, [object Object] and [object Object] are used to convert the data format from ND into NZ. Non-contiguous tensors support only the transpose scenario.

  • Ascend 950PR/Ascend 950DT:

    • The input x2 supports only the ND format. (The current version supports only two-dimensional inputs.)

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: aclnnQuantMatmulAllReduceV2 implements non-deterministic computation by default. You can set the HCCL_DETERMINISTIC environment variable to true to enable deterministic computation.
  • Ascend 950PR/Ascend 950DT: [object Object] defaults to a deterministic implementation.
• MC2 is disabled in incremental generation scenarios but enabled in full generation scenarios.
• [object Object] can be 2D (m, k) or 3D (b, s, k) and cannot be empty. [object Object] must be 2D and cannot be empty. (k, n), where the k axis meets the input parameter requirements of the MatMul operator. The k axes of [object Object] and [object Object] must be equal.
• The value of m cannot exceed 2147483647. The size of the last dimension of [object Object] (k) and [object Object] (k when transposed and n when not transposed) cannot exceed 65535.
• If [object Object] is not empty, its shape is (n). If [object Object] is not empty, its shape is the same as that of [object Object].
• When the shape of input [object Object] is (b, s, k), [object Object] shape is (b, s, n). When the shape of input [object Object] is (m, k), [object Object] shape is (m, n).
• The passed [object Object], [object Object], [object Object], or [object Object] cannot be a null pointer.
• The data types and formats of [object Object], [object Object], [object Object], [object Object], [object Object] (when not empty), and [object Object] (when not empty) must be supported.
• If [object Object] is of FLOAT16 type, the type of [object Object] is INT64 or UINT64 if [object Object] is empty, or FLOAT32 if [object Object] is not empty.
• If [object Object] is of BFLOAT16 type, the type of [object Object] is BFLOAT16.
• If the shape of [object Object] is (b, s, k), the shape of [object Object] is (b × s). If the shape of [object Object] is (m, k), the shape of [object Object] is (m).
• Only the all-mesh networking of HCCS links is supported.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: 1, 2, 4, or 8 ranks are supported.
  • Ascend 950PR/Ascend 950DT: 1, 2, 4, 8, 16, 32, or 64 cards are supported.
• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: supports only one communication domain for MC2 operators within a model.
• Support for empty tensors:
  • Empty tensors are not supported.

[object Object]

The following example is for reference only. For details, see .

Note: This sample code calls some HCCL collective communication library APIs: HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT:

  [object Object]