[object Object][object Object][object Object]undefined[object Object]

• Description: Fuses [object Object] communication, [object Object] (to ensure contiguous memory addresses after communication), [object Object], [object Object], and [object Object] computation using a communication-before-computation sequence. It supports K-C quantization, K-C dynamic quantization, and mx .

• The calculation formula is as follows: Assume that the input shape of x1 is (BS, H), the input shape of x1Scale in the mx quantization scenario is (BS, ceil(H/64), 2), and rankSize is the number of NPUs.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • K-C quantization scenario:

      $$commOut = AlltoAll(x1.view(rankSize, BS/rankSize, H)) \\ permutedOut = commOut.permute(1, 0, 2).view(BS/rankSize, rankSize*H) \\ output_{quant} = x1 @ x2 \\ output = output_{quant} \times x1_{scale} \times x2_{scale} \\ output = output + bias$$

    • K-C dynamic quantization scenario:

      $$commOut = AlltoAll(x1.view(rankSize, BS/rankSize, H)) \\ permutedOut = commOut.permute(1, 0, 2).view(BS/rankSize, rankSize*H) \\ x1_{quant}, x1_{scale} = Quant(permutedOut) \\ output_{quant} = x1_{quant} @ x2 \\ output = output_{quant} \times x1_{scale} \times x2_{scale} \\ output = output + bias$$

  • Ascend 950PR/Ascend 950DT:

    • K-C dynamic quantization scenario:

      $$commOut = AlltoAll(x1.view(rankSize, BS/rankSize, H)) \\ permutedOut = commOut.permute(1, 0, 2).view(BS/rankSize, rankSize*H) \\ dynQuantX1, dynQuantX1Scale = dynamicQuant(permutedOut) \\ output = (dynQuantX1@x2 + bias) \times dynQuantX1Scale \times x2Scale$$

    • mxQuantization scenario:

      $$commOut = AlltoAll(x1.view(rankSize, BS/rankSize, H)) \\ permutedOut = commOut.permute(1, 0, 2).view(BS/rankSize, rankSize*H) \\ commX1Scale = AlltoAll(x1Scale.view(rankSize, BS/rankSize, ceil(H/64), 2)) \\ permuteX1Scale = commX1Scale.permute(1, 0, 2, 3) \\ permutedX1Scale = permuteX1Scale.view(BS/rankSize, ceil(H/64)*rankSize, 2) \\ output = (permutedOut* permutedX1Scale)@(x2* x2Scale) + bias$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• ​Parameters:

  [object Object]

  The mapping between the enumerated values of x1QuantMode, x2QuantMode, and commQuantMode and the is as follows:

  • 0: no quantization
  • 1: pertensor
  • 2: perchannel
  • 3: pertoken
  • 4: pergroup
  • 5: perblock
  • 6: mxQuant
  • 7: per-token dynamic quantization

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing is supported by default.
• The number of NPUs (rankSize) varies depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: 2, 4, or 8 NPUs are supported.
  • Ascend 950PR/Ascend 950DT: 2, 4, 8, or 16 NPUs are supported.
• The variable BS used in the shape in the parameter description must be exactly divided by rankSize.
• The values of BS and N cannot exceed 2147483647 (INT32_MAX). The value of BS cannot be less than 2, and the value of N cannot be less than 1.
• Empty tensors are not supported.
• The support for non-contiguous tensors varies depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Non-contiguous tensors are not supported.
  • Ascend 950PR/Ascend 950DT: Only x2 can be a non-contiguous tensor. Other non-contiguous tensors are not supported.
• The input x1, x2, x2Scale, and output are not null pointers, and
  • Ascend 950PR/Ascend 950DT: In the case of x1QuantMode being pertoken dynamic quantization, x1ScaleOptional cannot be passed.
• The data types, dimensions, and quantization modes of the operator's inputs and outputs vary depending on the device model:
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • Quantization mode:

      • Currently, the left matrix supports perToken quantization and perToken dynamic quantization (x1QuantMode = 3 or 7), and the right matrix supports perChannel quantization (x2QuantMode = 2).

    • Type constraints:

      • The data types of x1 and alltoAllOutOptional must be the same.

      • If the int32 type is used for x1, x2, and alltoallout, the data is considered as eight packed int4s and will be reinterpreted as int4.

      • For A16W8 and A16W4, in the smoothQuant scenario, the data type of x1ScaleOptional must be the same as that of x1.

      • For A16W8, the supported data type combinations of x1, x2, biasOptional, and output are as follows:

        [object Object]undefined

      For * A16W4, the supported data type combinations of x1, x2, biasOptional, and output are as follows:

      [object Object]undefined

      For * A4W4, x1ScaleOptional supports only FLOAT32. The supported data type combinations of x1, x2, biasOptional, and output are as follows:

      [object Object]undefined

    • Dimension constraints:

      • For A16W8, rankSize x H must be exactly divided by 16. The value range of rankSize x H is [1, 35000].
      • For A16W4, rankSize x H must be exactly divided by 16. N must be an even number. The value range of rankSize x H is [1, 35000].
      • For A4W4, both H and N must be even numbers. The value range of rankSize x H is [1, 35000].
  • Ascend 950PR/Ascend 950DT:

    • Quantization mode:

      • Currently, the following modes are supported: K-C dynamic quantization, left matrix perToken dynamic quantization (x1QuantMode=7), right matrix perChannel quantization (x2QuantMode=2), and mx quantization. In mx quantization, the left matrix is quantized (x1QuantMode=6) and the right matrix is quantized (x2QuantMode=6).

    • Type constraints:

      • The data types of x1 and alltoAllOutOptional must be the same.
      • x1QuantDtype takes effect in the K-C dynamic quantization scenario. The value 35 (aclDataType.ACL_FLOAT8_E5M2) or 36 (aclDataType.ACL_FLOAT8_E4M3FN) can be configured. In other quantization scenarios, the configuration does not take effect.
      • biasOptional can be empty.
      • The supported data type combinations of the input and output are as follows:

        • K-C dynamic quantization:

          [object Object]undefined

        • mxQuantize:

          [object Object]undefined

    • Dimension constraints:

      • The value range of rankSize * H is [1, 65535].
      • In the mx quantization scenario, H must be exactly divided by 64.
      • In the mx quantization scenario, x2 must be transposed, the shape is (H*rankSize, N), and transposeX2 is True.
• MC2 operators cannot be called concurrently, nor can different MC2 operators.
• Inter-super node communication is not supported. Only intra-super node communication is supported.

[object Object]

The following example is for reference only. For details, see .

Note: This sample code calls some HCCL collective communication library APIs, including HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

• Ascend 950PR/Ascend 950DT:

  [object Object]