[object Object][object Object][object Object]undefined[object Object]

• Description: Fuses quantized matrix multiplication (MatMul) computation, data transposition (Permute) to ensure a contiguous memory data layout after communication, and AlltoAll collective communication. Computation is performed before communication. K-C quantization and mx are supported.

• Formulas: (Assume that the shape of [object Object] is [object Object], the shape of [object Object] is [object Object], and [object Object] indicates the number of NPU processors.)

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • K-C quantization scenario:$$computeOut = (x1 @ x2) * x1Scale * x2Scale + bias \\ permutedOut = computeOut.view(BS, rankSize, H2 / rankSize).permute(1, 0, 2) \\ output = AlltoAll(permutedOut).view(rankSize * BS, H2 / rankSize)$$

  • Ascend 950PR/Ascend 950DT:

    • K-C quantization scenario:

      $$computeOut = (x1 @ x2 + bias) * x1Scale * x2Scale \\ permutedOut = computeOut.view(BS, rankSize, H2 / rankSize).permute(1, 0, 2) \\ output = AlltoAll(permutedOut).view(rankSize * BS, H2 / rankSize)$$

    • mx quantization scenario:

      $$computeOut = (x1* x1Scale)@(x2* x2Scale) + bias \\ permutedOut = computeOut.view(BS, rankSize, H2 / rankSize).permute(1, 0, 2) \\ output = AlltoAll(permutedOut).view(rankSize * BS, H2 / rankSize)$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• ​Parameters:

  [object Object]

  The mapping between the enumerated values of x1QuantMode, x2QuantMode, and commQuantMode and the is as follows:

  • 0: no quantization
  • 1: pertensor
  • 2: perchannel
  • 3: pertoken
  • 4: pergroup
  • 5: perblock
  • 6: mxQuant
  • 7: per-token dynamic quantization

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing is supported by default.
• The number of NPUs (rankSize) varies depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: 2, 4, or 8 NPUs are supported.
  • Ascend 950PR/Ascend 950DT: 2, 4, 8, or 16 NPUs are supported.
• The variable H2 used in the shape must be exactly divided by the number of NPUs.
• The values of BS*rankSize and H2 cannot exceed 2147483647 (INT32_MAX). The value of BS cannot be less than 1, and the value of H2 cannot be less than 2.
• Empty tensors are not supported.
• The support for non-contiguous tensors varies depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Non-contiguous tensors are not supported.
  • Ascend 950PR/Ascend 950DT: Only x2 can be a non-contiguous tensor. Other non-contiguous tensors are not supported.
• The input x1, x2, x1Scale, x2Scale, and output are not null pointers, and
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: biasOptional cannot be a null pointer.
• The data types, dimensions, and quantization modes of the operator's input and output vary depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • Quantization mode:

      • Currently, the following modes are supported: K-C quantization, per-token quantization of the left matrix (x1QuantMode=3), and per-channel quantization of the right matrix (x2QuantMode=2).
      • The bias is added after quantization.

    • Type constraints:

      • The supported combinations of input and output data types are as follows:

        • K-C quantization:

          [object Object]undefined

    • Dimension constraints:

      • The H1 range is only [1, 65535].
  • Ascend 950PR/Ascend 950DT:

    • Quantization mode:

      • Currently, the following modes are supported: K-C quantization, left matrix per-token quantization (x1QuantMode = 3), right matrix per-channel quantization (x2QuantMode = 2), and mx quantization, left matrix mx quantization (x1QuantMode = 6), right matrix mx quantization (x2QuantMode = 6).
      • The bias is added before quantization.

    • Type constraints:

      • biasOptional can be empty.
      • The supported input/output data type combinations are as follows:

        • K-C quantization:

          [object Object]undefined

        • mx quantization:

          [object Object]undefined

    • Dimension constraints:

      • The H1 range supports only [1, 65535].
      • In the mx quantization scenario, x2 must be transposed, the shape is (H2, H1), and transposeX2 is True.
• MC2 operators cannot be called concurrently, nor can different MC2 operators.
• Inter-super node communication is not supported. Only intra-super node communication is supported.

[object Object]

The following example is for reference only. For details, see .

Note: In this example, some HCCL collective communication library APIs are called, including HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

• Ascend 950PR/Ascend 950DT:

  [object Object]