[object Object][object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication, supporting non-uniform matrix dimension sizes across multiple groups. The basic function is matrix multiplication, for example, $y_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups and $m_i$, $k_i$, and $n_i$ define the shapes for each group. The following four scenarios are supported based on the tensor count of $x$, $weight$, and $y$:

  • Multi-tensor $x$, $weight$, and $y$. That is, the tensors of each group are independent.
  • Single-tensor $x$, multi-tensor $weight$ and $y$. In this case, use the optional parameter [object Object] to define the row-wise grouping of $x$. For example, [object Object] indicates that the first 10 rows of $x$ participate in the multiplication of the first group of matrices.
  • Multi-tensor $x$ and $weight$, single-tensor $y$. In this case, products of each matrix group multiplication are stored contiguously within a single tensor.
  • Single-tensor $x$ and $y$, multi-tensor $weight$. This is a hybrid configuration combining the preceding two cases.

  Note: "Single-tensor" means that tensors of all groups in a tensor list are concatenated into one tensor along the M-axis.

• Formula:

  • Non-quantization scenario:
  $$y_i=x_i\times weight_i + bias_i$$
  • Quantization scenario:
  $$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$
  • Dequantization scenario:
  $$y_i=(x_i\times weight_i + bias_i) * scale_i$$
  • Fake-quantization scenario:
  $$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  • x (aclTensorList*, computation input): required parameter, aclTensorList on the device, $x$ in the formula. The can be ND, and the maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, or INT8.
  • weight (aclTensorList*, computation input): required parameter, aclTensorList on the device, $weight$ in the formula. The can be ND, and the maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, or INT8.
  • biasOptional (aclTensorList*, computation input): optional parameter, aclTensorList on the device, $bias$ in the formula. The can be ND, and the length is the same as that of [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, FLOAT32, or INT32.
  • scaleOptional (aclTensorList*, computation input): optional parameter, aclTensorList on the device, indicating the scale factor for quantization parameters. The can be ND, and the length is the same as that of [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be UINT64.
  • offsetOptional (aclTensorList*, computation input): optional parameter, aclTensorList on the device, indicating the offset for quantization parameters. The can be ND, and the length is the same as that of [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT32.
  • antiquantScaleOptional (aclTensorList*, computation input): optional parameter, aclTensorList on the device, indicating the scale factor for fake-quantization parameters. The can be ND, the length is the same as that of [object Object], and the data type can be FLOAT16 or BFLOAT16.
  • antiquantOffsetOptional (aclTensorList*, computation input): optional parameter, aclTensorList on the device, indicating the offset for fake-quantization parameters. The can be ND, the length is the same as that of [object Object], and the data type can be FLOAT16 or BFLOAT16.
  • groupListOptional (aclIntArray*, computation input): optional parameter, aclIntArray type on the host, indicating the Matmul indexing for the input and output along the M-axis. The data type can be INT64, the can be ND, and the length is the same as that of [object Object]. Note that when the length of the TensorList in the output is 1, the last value in [object Object] constrains the valid portion of the output data. Any portion not specified in [object Object] will not be updated.
  • splitItem (int64_t, computation input): integer type, indicating whether tensor splitting is required for the output. [object Object] or [object Object] indicates multi-tensor, and [object Object] or [object Object] indicates single-tensor.
  • y (aclTensorList*, computation output): aclTensorList on the device, $y$ in the formula. The can be ND, and the maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, INT8, or FLOAT32.
  • workspaceSize (uint64_t*, output): size of the workspace to be allocated on the device.
  • executor (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Return

  [object Object] status code. For details, see .

[object Object]

• Parameters

  • workspace (void*, input): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling [object Object].
  • executor (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • stream (aclrtStream, input): stream for executing the task.

• Return

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • The following input types are supported in non-quantization scenarios:

    • [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT16
    • [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16
    • [object Object]: FLOAT32; [object Object]: FLOAT32; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT32

  • The following input types are supported in quantization scenarios:

    • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: INT8

  • The following input types are supported in fake-quantization scenarios:

    • [object Object]: FLOAT16; [object Object]: INT8; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: FLOAT16
    • [object Object]: BFLOAT16; [object Object]: INT8; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: BFLOAT16

  • If [object Object] is passed, it must be a non-negative ascending array, and its length cannot be 1.

  • The following scenarios are supported: "S" stands for single-tensor, and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

    [object Object]undefined

  • The size of the last dimension for each tensor in [object Object] and [object Object] should be less than 65536. The last dimension of $x_i$ refers to the K-axis when [object Object] is false or the M-axis when [object Object] is true. The last dimension of $weight_i$ refers to the N-axis when [object Object] is false or the K-axis when [object Object] is true.

  • The size of each dimension for every tensor in [object Object] and [object Object], after 32-byte alignment, should be less than the maximum value of INT32 (2147483647).

[object Object]

The following example is for reference only. For details, see .