aclnnSwiGluQuantV2

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Adds a quant operation after the SwiGlu activation function to perform SwiGluQuant computation on input x, supporting int8 or int4 quantized output.
Description of API differences: Compared with the aclnnSwiGluQuant API, aclnnSwiGluQuantV2 newly supports the cumsum mode and count mode for groupIndexOptional, with different modes controlled by groupListType; newly supports non-MoE scenarios (groupIndexOptional passes null); and newly supports int8 or int4 quantized output yOut, with different quantized output data types controlled by dstType.
Operator support scope: Currently, SwiGluQuant supports MoE scenarios (with groupIndexOptional passed) and non-MoE scenarios (groupIndexOptional passed as null). The inputs x and group_index of SwiGluQuant come from the outputs of the GroupedMatMul operator and MoeInitRouting. The group_index parameter enables MoE group-based dynamic quantization, static per-tensor quantization, and static per-channel quantization.
Dynamic quantization formula for MoE scenarios:
$Act = SwiGlu(x) = Swish(A)*B \\ Y_{tmp}^0 = Act[0\colon g[0],\colon] * smooth\_scales[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))$ $Y = Cast(Mul(Y_{tmp}, Scale))$
Where A represents the first half of input x, B represents the second half of input x, g represents group_index, and G is the number of groups in group_index. For int8 quantization, $dstTypeScale = 127$ (127 is the maximum value of int8); for int4 quantization, $dstTypeScale = 7$ (7 is the maximum value of int4).
Static quantization formula for MoE scenarios:
$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}^0 = Act(0\colon g[0],\colon) * smooth\_scales[0\colon g[0],\colon] + offsets[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon] + offsets[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\$ $Y = Cast(Y_{tmp})$
Where A represents the first half of input x, B represents the second half of input x, g represents group_index, and G is the number of groups in group_index.
Non-MoE scenario (groupIndexOptional is passed null) dynamic quantization formula:
$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act* smooth\_scales(0,\colon)\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))$ $Y = Cast(Mul(Y_{tmp}, Scale))$
Where A represents the first half of input x, B represents the second half of input x. For int8 quantization, $dstTypeScale = 127$ (127 is the maximum value of int8); for int4 quantization, $dstTypeScale = 7$ (7 is the maximum value of int4).
Non-MoE scenario (groupIndexOptional is passed null) static quantization formula:
$Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act * smooth\_scales(0,\colon) + offsets(0,\colon) \\$ $Y = Cast(Y_{tmp})$
Where A represents the first half of input x, and B represents the second half of input x.

Function Prototype

Each operator is divided into a two-phase API. The "aclnnSwiGluQuantV2GetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnSwiGluQuantV2" API is called to perform the computation.

aclnnStatus aclnnSwiGluQuantV2GetWorkspaceSize(
  const aclTensor *x,
  const aclTensor *smoothScalesOptional,
  const aclTensor *offsetsOptional,
  const aclTensor *groupIndexOptional,
  bool             activateLeft,
  char            *quantModeOptional,
  int64_t          groupListType,
  int64_t          dstType,
  const aclTensor *yOut,
  const aclTensor *scaleOut,
  uint64_t        *workspaceSize,
  aclOpExecutor  **executor)

aclnnStatus aclnnSwiGluQuantV2(
  void            *workspace,
  uint64_t         workspaceSize,
  aclOpExecutor   *executor,
  aclrtStream      stream)

aclnnSwiGluQuantV2GetWorkspaceSize

Parameters:

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
x (aclTensor*)	Input	The input data to be processed, i.e., x in the formula.	The last dimension of x must be a multiple of 2, and the number of dimensions of x must be greater than 1. Currently only supports the last dimension length of x not exceeding 8192. When dstType is set to 29 (indicating yOut output as INT4 quantization), the last dimension of x must be a multiple of 4.	FLOAT16, BFLOAT16, FLOAT	ND	-	×
smoothScalesOptional（aclTensor*）	Input	Smooth scales for quantization, i.e., smooth_scales in the formula.	Empty tensors are not supported. Shape supports [G, N] and [G, ], where G represents the number of groupIndex groups and N is half the size of the last dimension of the compute input x.	FLOAT	ND	-	×
offsetsOptional (aclTensor*)	Input	The offsets in the formula.	This parameter does not take effect in the dynamic quantization scenario. The user can pass a null pointer. In the static quantization scenario: the data type supports FLOAT. In per_channel mode, the shape supports [G, N]. In per_tensor mode, the shape supports [G, ], and the data type and shape must remain consistent with smoothScalesOptional.	FLOAT	ND	-	×
groupIndexOptional (aclTensor*)	Input	Group index required for MoE grouping, i.e., group_index in the formula.	The shape supports [G, ]. Elements in group_index must be non-decreasing, and the maximum value must not exceed the product of all dimensions of input x except the last dimension. The value of G must not exceed the product of all dimensions of input x except the last dimension.	INT32	ND	-	×
activateLeft (bool)	Input	Indicates whether the left matrix participates in the computation. The user must pass this parameter.	-	-	-	-	-
quantModeOptional (char*)	Input	Compute input.	The user must pass a parameter. "static" indicates static quantization, "dynamic" indicates dynamic quantization, and "dynamic_msd" indicates dynamic MSD quantization. Currently only supports "dynamic" dynamic quantization and "static" static quantization. Static quantization only supports per-tensor quantization and per-channel quantization.	-	-	-	-
groupListType (int64_t)	Input	Compute input.	The user must pass a parameter. 0 indicates cumsum mode and 1 indicates count mode. Currently only supports 0, which indicates cumsum mode.	-	-	-	-
dstType (int64_t)	Input	Compute input.	The user must pass a parameter. 2 indicates that yOut is int8 quantized output, and 29 indicates that yOut is int4 quantized output. Currently only supports 2 and 29, with a default value of 2.	-	-	-	-
yOut (aclTensor*)	Output	Output tensor.	The last dimension size of the computed output yOut is half of the last dimension size of the compute input x, and the remaining dimensions remain consistent with x.	INT8, INT4	ND	-	×
scaleOut (aclTensor*)	Output	Output tensor.	The shape of the computed output scaleOut, compared with the compute input x, has no last dimension, and the remaining dimensions remain consistent with the compute input x.	FLOAT	ND	-	×
workspaceSize (uint64_t*)	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
executor (aclOpExecutor**)	Output	Returns the op executor, which includes the computation flow of the operator.	-	-	-	-	-

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The passed x or yOut is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The data type of the input or output is not in the supported range.
		The dimension of the input or output parameter is not in the supported range.
		quantModeOptional is not in the specified value range.
		groupListType is not in the specified value range.
		dstType is not in the specified value range.

aclnnSwiGluQuantV2

Parameters:

Parameter	Input/Output	Description
workspace	Input	The workspace memory address allocated on the Device side.
workspaceSize	Input	The workspace size allocated on the Device side, obtained from the first-phase API aclnnSwiGluQuantV2GetWorkspaceSize.
executor	Input	Operator executor that includes the operator computation flow.
stream	Input	Specifies the stream for task execution.

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnSwiGluQuantV2 defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_swi_glu_quant_v2.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate: initialize resources.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to allocate device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Copy data from the host side to the device memory by calling aclrtMemcpy.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Create an aclTensor by calling the aclCreateTensor API.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API.
  // Set deviceId based on your actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct inputs and outputs. Customize the construction based on the API interface.
  std::vector<int64_t> xShape = {3, 32};
  std::vector<int64_t> smoothScalesShape = {2, 16};
  std::vector<int64_t> groupIndexShape = {2};
  std::vector<int64_t> outShape = {3, 16};
  std::vector<int64_t> scaleShape = {3};
  void* xDeviceAddr = nullptr;
  void* smoothScalesDeviceAddr = nullptr;
  void* groupIndexDeviceAddr = nullptr;
  void* outDeviceAddr = nullptr;
  void* scaleDeviceAddr = nullptr;
  aclTensor* x = nullptr;
  aclTensor* smoothScales = nullptr;
  aclTensor* groupIndex = nullptr;
  aclTensor* out = nullptr;
  aclTensor* scale = nullptr;
  std::vector<float> xHostData = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
                                  23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
                                  43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62,
                                  63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,
                                  83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95};
  std::vector<float> smoothScalesHostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
                                      1, 1, 1, 1, 1, 1, 1, 1};
  std::vector<float> groupIndexHostData = {1, 3};
  std::vector<int8_t> outHostData = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                                      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
  std::vector<float> scaleHostData = {0, 0, 0};

  // Create the x aclTensor.
  ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_BF16, &x);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
   // Create the scale aclTensor.
  ret = CreateAclTensor(smoothScalesHostData, smoothScalesShape, &smoothScalesDeviceAddr, aclDataType::ACL_FLOAT, &smoothScales);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
   // Create the groupIndex aclTensor.
  ret = CreateAclTensor(groupIndexHostData, groupIndexShape, &groupIndexDeviceAddr, aclDataType::ACL_INT32, &groupIndex);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the out aclTensor.
  ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT8, &out);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the scale aclTensor.
  ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // 3. Call the CANN operator library API. Replace with the specific API name.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  // Call the first-phase API of aclnnSwiGluQuantV2.
  ret = aclnnSwiGluQuantV2GetWorkspaceSize(x, smoothScales, nullptr, groupIndex, false, "dynamic", 0, 2, out, scale, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnSwiGluQuantV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Allocate device memory based on the workspaceSize computed by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the second-phase API of aclnnSwiGluQuantV2.
  ret = aclnnSwiGluQuantV2(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnSwiGluQuantV2 failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronously wait for task execution to finish.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device-side memory to the host side. Modify based on the specific API definition.
  auto size = GetShapeSize(outShape);
  std::vector<int8_t> resultData(size, 0);
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
  }
  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(x);
  aclDestroyTensor(smoothScales);
  aclDestroyTensor(groupIndex);
  aclDestroyTensor(out);
  aclDestroyTensor(scale);
  // 7. Release device resources. Modify based on the specific API definition.
  aclrtFree(xDeviceAddr);
  aclrtFree(smoothScalesDeviceAddr);
  aclrtFree(groupIndexDeviceAddr);
  aclrtFree(outDeviceAddr);
  aclrtFree(scaleDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}