[object Object]

[object Object][object Object]undefined[object Object]

• API function: This API is compatible with the functions supported by aclnnQuantMatmulAllReduce, aclnnQuantMatmulAllReduceV2, and aclnnQuantMatmulAllReduceV3.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: No new features are added.
  • Ascend 950PR/Ascend 950DT: The perblock, pertile, and mxfp quantization modes are added. The x1 and x2 inputs support the data type of FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8 and FLOAT4_E2M1.

• Formula:

  • Formula 1 and formula 2 or 3 for enabling low-bit communication:

    When x1 and x2 are of type INT8 and commQuantScale1Optional and commQuantScale2Optional are not empty:

    $$matmulAddOutput = (x2ScaleOptional * x1ScaleOptional * (x1_{int8}@x2_{int8} + biasOptional_{int32}) + x3Optional);$$ $$alltoallOutput_{int8} = AllToAll(matmulAddOutput / commQuantScale1Optional);$$ $$reduceSumOutput_{int8} = (add(alltoallOutput_{int8}) * (commQuantScale1Optional / commQuantScale2Optional));$$ $$output = (AllGather(reduceSumOutput_{int8}) * commQuantScale2Optional);$$

  • Formula 2 (per-channel quantization and per-tensor quantization):

    x1 and x2 are of type INT8. x1ScaleOptional is not supported. x2ScaleOptional is of type INT64 or UINT64. biasOptional is of type INT32 (optional). out is of type BFLOAT16 or FLOAT16.

    $$output = AllReduce((x1@x2 + biasOptional) * x2ScaleOptional + x3Optional)$$

  • Formula 3 (per-token per-channel quantization and per-token per-tensor quantization):

    x1 and x2 are of type INT8. x1ScaleOptional is of type FLOAT32. x2Scale is of type FLOAT32 or BFLOAT16. biasOptional is of type INT32 (optional). out is of type FLOAT16 or BFLOAT16.

    $$output = AllReduce((x1@x2 + biasOptional) * x2ScaleOptional * x1ScaleOptional + x3Optional)$$

  • Formula 4 (MXFP quantization):

    x1 and x2 are of type FLOAT4_E2M1/FLOAT8_E4M3FN/FLOAT8_E5M2. x1ScaleOptional is of type FLOAT8_E8M0. x2ScaleOptional is of type FLOAT8_E8M0. biasOptional is of type FLOAT32 (optional). out is of type FLOAT16, BFLOAT16, or FLOAT32.

    $$output = AllReduce((x1* x1ScaleOptional)@(x2* x2ScaleOptional) + biasOptional + x3Optional)$$

  • Formula 5 (per-channel quantization and per-tensor quantization):

    x1 and x2 are of type FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, x2ScaleOptional is of type UINT/INT64, the optional bias is of type FLOAT32, and out is of type FLOAT16/BFLOAT16/FLOAT32.

  $$output = AllReduce((x1@x2 + biasOptional) * x2ScaleOptional + x3Optional)$$

  • Formula 6 (per-token-per-channel quantization and per-token-per-tensor quantization):

    x1 and x2 are of type FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, x1ScaleOptional and x2ScaleOptional are of type FLOAT32, the optional bias is of type FLOAT32, and out is of type FLOAT16/BFLOAT16/FLOAT32.

    $$output = AllReduce((x1@x2 + biasOptional) * x2ScaleOptional * x1ScaleOptional + x3Optional)$$

  • Formula 7 (per-block-per-block quantization):

    x1 and x2 are of type FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, x1ScaleOptional is of type FLOAT32, x2Scale is of type FLOAT32, and there is no biasOptional. When x1 is (a0, a1) and x2 is (b0, b1), x1ScaleOptional is (ceilDiv(a0, 128), ceilDiv(a1, 128)), x2Scale is (ceilDiv(b0, 128), ceilDiv(b1, 128)), and out is of type FLOAT16/BFLOAT16/FLOAT32.

    $$output_{pq} = AllReduce(\sum_{0}^{\left \lfloor \frac{k}{128} \right \rfloor} (x1_{pr}@x2_{rq}*(x1ScaleOptional_{pr}*x2Scale_{rq})) + x3Optional)$$

  • Formula 8 (enabling low-bit communication and per-tile quantization):

    x1 and x2 are of type FLOAT8_E4M3FN/FLOAT8_E5M2, x1ScaleOptional is of type FLOAT32, x2Scale is of type FLOAT32, biasOptional is of type FLOAT32 (optional), commQuantMode is 1, and out is of type FLOAT16/BFLOAT16/FLOAT32.

    $$matmulAddOutput_{fp32} = (x2ScaleOptional * x1ScaleOptional * (x1_{fp8}@x2_{fp8} + biasOptional_{fp32}) + x3Optional);$$ $$scaleOut_{fp32} = (matmulAddOutput_{fp32} / (reduceMax(abs(matmulAddOutput_{fp32})) / FP32\_MAX));$$ $$quantOutput_{fp8} = (append((matmulAddOutput_{fp32} * scaleOut{fp32})@scaleOut_{fp32}));$$ $$alltoallOutput_{fp8} = (AllToAll(quantOut_{fp8}));$$ $$dequantOutput_{fp32} = (alltoallOutput_{fp8} / scaleOut_{fp32});$$ $$reduceSumOutput_{fp32} = (reduceSum(dequantOutput_{fp32}));$$ $$preAllGatherQuantScale_{fp32} = (reduceSumOutput_{fp32} / (reduceMax(abs(reduceSumOutput_{fp32})) / FP8\_MAX));$$ $$preAllGatherQuantOutput_{fp8} = (append((reduceSumOutput_{fp32} * preAllGatherQuantScale_{fp32})@preAllGatherQuantScale_{fp32}));$$ $$allGatherOutput_{fp8} = (AllGather(preAllGatherQuantOutput_{fp8}));$$ $$output = (cast(allGatherOutput_{fp32} / preAllGatherQuantScale_{fp32}));$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: aclnnQuantMatmulAllReduceV4 is implemented in non-deterministic mode by default. You can enable deterministic computing by setting the environment variable HCCL_DETERMINISTIC to true.
  • Ascend 950PR/Ascend 950DT: aclnnQuantMatmulAllReduceV4 is implemented in deterministic mode by default.
• MC2 is disabled in incremental scenarios but enabled in full scenarios.
• The input x1 can be 2D or 3D, and its shape is (b, s, k) or (m, k). x2 must be 2-dimensional. Its shape is (k, n). The k axes meet the input parameter requirements of the mm operator and are equal.
• The value of [object Object] cannot exceed [object Object]. The last dimension of [object Object] is [object Object], and that of [object Object] is [object Object] in the transpose scenario or [object Object] in the non-transpose scenario. The size of the last dimensions of [object Object] and [object Object] cannot exceed [object Object].
• The input x1, x2, x2Scale, or output is not a null pointer.
• The data types and formats of [object Object], [object Object], [object Object], [object Object], [object Object] (when not empty), and [object Object] (when not empty) must be supported.
• If the passed [object Object] and [object Object] are not null pointers, their shapes must be the same, their types must be the same as the operator output type, and the inputs for each rank must be the same.
• Only the all-mesh networking of HCCS links is supported.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: 1, 2, 4, and 8 ranks are supported.
  • Ascend 950PR/Ascend 950DT: 1, 2, 4, 8, 16, 32, and 64 cards are supported.
• The merged compute and communication (MC2) operators in a model support only the same communicator.
• The performance gain of INT8 and FP8 low-bit communication is available only when the communication bound is reached. In the case of the computation bound, you are advised not to enable INT8 or FP8 low-bit communication, that is, you are advised not to set commQuantScale1 and commQuantScale2 and set commQuantMode to 0. (Note: INT8 low-bit communication refers to the scenario where the input is int8 and commQuantScale1Optional and commQuantScale2Optional are enabled. FP8 low-bit communication refers to the scenario where the input is FLOAT8_E4M3FN/FLOAT8_E5M2 and commQuantMode is set to 1.)
• Support for empty tensors:
  • Empty tensors are not supported.
• Restrictions on groupSize:
  • The value of groupSize is valid only when both x1Scale and x2Scale are 2D or higher-dimensional data. In other scenarios, 0 needs to be passed.
  • The input groupSize is decomposed into groupSizeM, groupSizeN, and groupSizeK according to the following formulas. If one or more of them are 0, groupSizeM, groupSizeN, and groupSizeK are reset based on the input shape of x1, x2, x1Scale, or x2Scale for calculation. Principle: Assume that groupSizeM = 0, indicating that the quantization group size in the m direction is inferred by the API. The inference formula is groupSizeM = m / scaleM (ensure that m is exactly divisible by scaleM). m is the same as that in the x1 shape, and scaleM is the same as that in the x1Scale shape.$$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$$

The following table describes the supported input and output data type combinations.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

• Ascend 950PR/Ascend 950DT:

  When int8 is used as the input, per-token-per-channel quantization and per-tensor-per-channel quantization are supported.

  [object Object]

  per-token-per-channel quantization && per-token-per-tensor quantization && per-block-per-block quantization

  [object Object]

  Per-channel quantization and per-tensor quantization

  [object Object]

  MXFP quantization

  [object Object]

[object Object]

The following example is for reference only. For details, see .

Note: In this sample code, some HCCL collective communication library APIs are called, including HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• For the [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT:

  [object Object]