aclnnDequantSwigluQuant

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	×
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Adds dequant and quant operations before and after the Swish Gated Linear Unit activation function to implement DequantSwigluQuant computation on x.
Formula:
$dequantOut = Dequant(x, weightScaleOptional, activationScaleOptional, biasOptional)$ $swigluOut = Swiglu(dequantOut)=Swish(A)*B$ $out = Quant(swigluOut, quantScaleOptional, quantOffsetOptional)$
where A represents the first half of dequantOut, and B represents the second half of dequantOut.

Function Prototype

Each operator is divided into a two-phase API. The "aclnnDequantSwigluQuantGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnDequantSwigluQuant" API is called to perform the computation.

aclnnStatus aclnnDequantSwigluQuantGetWorkspaceSize(
    const aclTensor *x,
    const aclTensor *weightScaleOptional,
    const aclTensor *activationScaleOptional,
    const aclTensor *biasOptional,
    const aclTensor *quantScaleOptional,
    const aclTensor *quantOffsetOptional,
    const aclTensor *groupIndexOptional,
    bool             activateLeft,
    char            *quantModeOptional,
    const aclTensor *yOut,
    const aclTensor *scaleOut,
    uint64_t        *workspaceSize,
    aclOpExecutor  **executor)

aclnnStatus aclnnDequantSwigluQuant(
    void          *workspace,
    uint64_t       workspaceSize,
    aclOpExecutor *executor,
    aclrtStream    stream)

aclnnDequantSwigluQuantGetWorkspaceSize

Parameters:

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
x (aclTensor*)	Input	Input data to be processed, i.e., x in the formula.	Shape is (N..., H). The last dimension must be a multiple of 2, and the dimension of x must be greater than 1.	FLOAT16, BFLOAT16, INT32	ND	2-8	x
weightScaleOptional (aclTensor*)	Input	Dequantization scale of weight, i.e., weightScaleOptional in the formula.	Atlas A2 training products/Atlas A2 inference products: shape is represented as [H], and the value H is consistent with the last dimension of x. Ascend 950PR/Ascend 950DT: shape is represented as [H] or [groupNum, H], and the value H is consistent with the last dimension of x. When groupIndexOptional is a null pointer, the shape is [H]; when groupIndexOptional is not a null pointer, the shape is [groupNum, H]. Optional parameter. Supports passing null pointer.	FLOAT	ND	1 or 2	x
activationScaleOptional (aclTensor*)	Input	Dequantization scale of the activation function, i.e., activationScaleOptional in the formula.	Atlas A2 training products/Atlas A2 inference products: shape is [N..., 1], where the last dimension is 1 and the remaining dimensions are consistent with x. Ascend 950PR/Ascend 950DT: shape is [N,...], with no more than 7 dimensions and no fewer than 1 dimension, one dimension fewer than x, and the shape is consistent with the corresponding dimensions of x. Optional parameter. Supports passing null pointer.	FLOAT	ND	1	x
biasOptional (aclTensor*)	Input	Bias of Matmul, i.e., biasOptional in the formula.	The shape supports 1D, represented as [H], and the value H is consistent with the last dimension of x. Optional parameter, supports passing null pointer.	FLOAT, FLOAT16, BFLOAT16, INT32	ND	1	x
quantScaleOptional (aclTensor*)	Input	Quantization scale, i.e., quantScaleOptional in the formula.	Atlas A2 training products/Atlas A2 inference products: When quantModeOptional is static, the shape is represented as shape[1]; when quantModeOptional is dynamic, the shape has one dimension and its value is half of the last dimension of x, represented as shape[H/2]. Optional parameter, supports passing null pointer. Ascend 950PR/Ascend 950DT: Only supports FLOAT, and only supports the scenario where quantModeOptional is dynamic. When quantModeOptional is dynamic, the shape is represented as [H/2] or [groupNum, H/2]. When groupIndexOptional is a null pointer, the shape is [H/2]; when groupIndexOptional is not a null pointer, the shape is [groupNum, H/2].	FLOAT, FLOAT16	ND	1 or 2	x
quantOffsetOptional (aclTensor*)	Input	Quantization offset, i.e., quantOffsetOptional in the formula.	Atlas A2 training products/Atlas A2 inference products: When quantModeOptional is static, the shape is 1D with a value of 1, represented as shape[1]; when quantModeOptional is dynamic, the shape is 1D with a value of half of the last dimension of x, represented as shape[H/2]. Optional parameter, supports passing null pointer. Ascend 950PR/Ascend 950DT: This parameter is not supported currently.	FLOAT	ND	-	x
groupIndexOptional (aclTensor*)	Input	The group_index required for MoE grouping.	Ascend 950PR/Ascend 950DT: Only supports INT64. The shape is [groupNum], where groupNum must be greater than or equal to 1. Optional parameter, supports passing null pointer. Optional parameter, supports passing null pointer.	INT32, INT64	ND	1	x
activateLeft (bool)	Input	Indicates whether to apply SwiGLU activation to the left half of the input.	When the value is false, activation is applied to the right half of the input.	-	-	-	-
quantModeOptional (char*)	Input	Indicates whether dynamic quantization or static quantization is used.	Atlas A2 training products/Atlas A2 inference products: "dynamic" and "static" are supported. Ascend 950PR/Ascend 950DT: Only "dynamic" is supported. Supports passing null. When a null pointer is passed, it defaults to "static".	-	-	-	-
yOut (aclTensor*)	Output	-	Ascend 950PR/Ascend 950DT: shape is [N..., H/2], and the last axis of yOut must be less than 5120.	INT8	ND	-	x
scaleOut (aclTensor*)	Output	-	Ascend 950PR/Ascend 950DT: The shape has no more than 7 dimensions and no fewer than 1 dimension, with shape [N,...] that is consistent with the shape of yOut after removing the last dimension.	FLOAT	ND	-	x
workspaceSize (uint64_t*)	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
executor (aclOpExecutor**)	Output	Returns the op executor, which includes the computation flow of the operator.	-	-	-	-	-

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The passed x, yOut, or scaleOut is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The data type of the input or output is not within the supported range.
		The parameter dimension of the input or output is not within the supported range.
		The shape of the input or output does not meet the constraint requirements.
		The input value does not meet the requirements.
ACLNN_ERR_INNER_TILING_ERROR	561002	The memory size of the input tensor exceeds the upper limit.

aclnnDequantSwigluQuant

Parameters:

Parameter	Input/Output	Description
workspace	Input	Address of the workspace memory allocated on the Device side.
workspaceSize	Input	Size of the workspace allocated on the Device side, obtained by the first-phase API aclnnDequantSwigluQuantGetWorkspaceSize.
executor	Input	Operator executor, which includes the computation process of the operator.
stream	Input	Specifies the stream for task execution.

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnDequantSwigluQuant defaults to a deterministic implementation.
Atlas A2 training products/Atlas A2 inference products:
- The last dimension of x must be a multiple of 2, and the dimension of x must be greater than 1.
- When quantModeOptional is static, quantScaleOptional and quantOffsetOptional are 1-dimensional with a value of 1. When quantModeOptional is dynamic, quantScaleOptional and quantOffsetOptional are 1-dimensional with a value equal to the last dimension of x divided by 2.
Ascend 950PR/Ascend 950DT:
- The dimension of input x corresponding to activateDim must be a multiple of 2, and the number of dimensions of x must be greater than 1.
- When the data type of input x is INT32, weightScaleOptional cannot be null. When the data type of input x is not INT32, weightScaleOptional is not allowed and a null pointer is passed.
- When the data type of input x is not INT32, activationScaleOptional is not allowed and a null pointer is passed.
- When the data type of input x is not INT32, biasOptional input is not allowed. Pass a null pointer.
- The last axis of output yOut does not exceed 5120.
Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products: The operator imposes an upper limit on the memory size of input tensors. Verification formula: weightScaleOptional tensor memory size + biasOptional tensor memory size + quantScaleOptional tensor memory size + quantOffsetOptional tensor memory size + (activationScaleOptional tensor memory size + scaleOut tensor memory size) / 40 + x tensor last dimension H memory size * 10 < 192 KB.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_dequant_swiglu_quant.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // (Fixed writing) Initialize AscendCL.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to allocate memory on the device.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Call aclrtMemcpy to copy the data on the host to the memory on the device.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Compute the strides of the contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call aclCreateTensor to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Fixed writing) Initialize the device and stream. For details, see the list of external AscendCL APIs.
  // Set the device ID in use.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the input and output based on the API.
  std::vector<int64_t> xShape = {2, 32};
  std::vector<int64_t> scaleShape = {1};
  std::vector<int64_t> offsetShape = {1};
  std::vector<int64_t> outShape = {2, 16};
  std::vector<int64_t> scaleOutShape = {2};
  void* xDeviceAddr = nullptr;
  void* scaleDeviceAddr = nullptr;
  void* offsetDeviceAddr = nullptr;
  void* outDeviceAddr = nullptr;
  void* scaleOutDeviceAddr = nullptr;
  aclTensor* x = nullptr;
  aclTensor* scale = nullptr;
  aclTensor* offset = nullptr;
  aclTensor* out = nullptr;
  aclTensor* scaleOut = nullptr;
  std::vector<float> xHostData = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
                                    23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
                                    43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63};
  std::vector<float> scaleHostData = {1};
  std::vector<float> offsetHostData = {1};
  std::vector<int8_t> outHostData = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
  std::vector<float> scaleOutHostData = {0, 0};

  // Create an x aclTensor.
  ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
   // Create a scale aclTensor.
  ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
   // Create an offset aclTensor.
  ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create an out aclTensor.
  ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT8, &out);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create scaleOut aclTensor.
  ret = CreateAclTensor(scaleOutHostData, scaleOutShape, &scaleOutDeviceAddr, aclDataType::ACL_FLOAT, &scaleOut);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // 3. Call the CANN operator library API, which needs to be replaced with the actual API.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  // Call the first-phase API of aclnnDequantSwigluQuant.
  ret = aclnnDequantSwigluQuantGetWorkspaceSize(x, nullptr, nullptr, nullptr, scale, offset, nullptr, false, "static", out, scaleOut, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnDequantSwigluQuantGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Allocate device memory based on the computed workspaceSize.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the second-phase API of aclnnDequantSwigluQuant.
  ret = aclnnDequantSwigluQuant(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnDequantSwigluQuant failed. ERROR: %d\n", ret); return ret);

  // 4. (Fixed writing) Wait until the task execution is complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device memory to the host. Modify the configuration based on the API definition.
  auto size = GetShapeSize(outShape);
  std::vector<int8_t> resultData(size, 0);
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
  }
  // 6. Release aclTensor and aclScalar. Modify the configuration based on the API definition.
  aclDestroyTensor(x);
  aclDestroyTensor(scale);
  aclDestroyTensor(offset);
  aclDestroyTensor(out);
  aclDestroyTensor(scaleOut);
  // 7. Release device resources. Modify the configuration based on the API definition.
  aclrtFree(xDeviceAddr);
  aclrtFree(scaleDeviceAddr);
  aclrtFree(offsetDeviceAddr);
  aclrtFree(outDeviceAddr);
  aclrtFree(scaleOutDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}