aclnnGroupNormSiluQuant

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	x
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Computes the group normalization of the input self, outputs the mean meanOut, the reciprocal of the standard deviation rstdOut, and the quantized result out of the SiLU output.
Formula:
- GroupNorm:
Let $E[x] = \bar{x}$ represent the mean of $x$ , and $Var[x] = \frac{1}{n} * \sum_{i=1}^n(x_i - E[x])^2$ represent the variance of $x$ , then
$\left\{ \begin{array} {rcl} groupNormOut& &= \frac{x - E[x]}{\sqrt{Var[x] + eps}} * \gamma + \beta \\ meanOut& &= E[x]\\ rstdOut& &= \frac{1}{\sqrt{Var[x] + eps}}\\ \end{array} \right.$
- SiLU:
$siluOut = \frac{groupNormOut}{1+e^{-groupNormOut}}$
- Quant:
$out = round(siluOut / quantScale)$

Function Prototype

Each operator is divided into a two-phase API. The "aclnnGroupNormSiluQuantGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnGroupNormSiluQuant" API is called to perform the computation.

aclnnStatus aclnnGroupNormSiluQuantGetWorkspaceSize(
    const aclTensor* self,
    const aclTensor* gammaOptional,
    const aclTensor* betaOptional,
    const aclTensor* quantScale,
    int64_t          group,
    double           eps,
    bool             activateSilu,
    aclTensor*       out,
    aclTensor*       meanOut,
    aclTensor*       rstdOut,
    uint64_t*        workspaceSize,
    aclOpExecutor**  executor);

aclnnStatus aclnnGroupNormSiluQuant(
    void *         workspace,
    uint64_t       workspaceSize,
    aclOpExecutor *executor,
    aclrtStream    stream)

aclnnGroupNormSiluQuantGetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
self	Input	x in the formula.	-	FLOAT16, BFLOAT16	ND	2-8, where the 0th dimension is N and the 1st dimension is C.	√
gammaOptional	Optional input.	γ in the formula.	Defaults to 1 when the element is empty. The data type must remain consistent with self, and the number of elements must be the same as the first dimension of the input self.	FLOAT16, BFLOAT16	ND	1	√
betaOptional	Optional input	β in the formula.	When left empty, the default value of the element is 1. The data type must remain consistent with self, and the number of elements must be the same as the first dimension of self.	FLOAT16, BFLOAT16	ND	1	√
quantScale	Input	quantScale in the formula.	The number of elements must be 1 or remain the same as the first dimension of the input self.	FLOAT32	ND	1	√
group	Input	Indicates that the first dimension of the input self is divided into group groups.	group must be divisible by the first dimension of self.	INT64	-	-	-
eps	Input.	eps in the formula.	eps must be greater than 0.	DOUBLE	-	-	-
activateSilu	Input	Whether to enable SiLU computation.	Only the enabled state is currently supported.	BOOL	-	-	-
out	Output	Quantized result, which is out in the formula.	-	INT8	ND	Same as self.	-
meanOut	Output	meanOut in the formula.	The data type remains consistent with self, and N in the shape remains consistent with the 0th dimension of self.	FLOAT16, BFLOAT16	ND	(N, group)	-
rstdOut	Output	rstdOut in the formula.	The data type remains consistent with self, and N in the shape remains consistent with the 0th dimension of self.	FLOAT16, BFLOAT16	ND	(N, group)	-
workspaceSize	Output	Returns the workspace size to be applied on the Device side.	-	-	-	-	-
executor	Output	Returns the operator executor, which includes the computation flow of the operator.	-	-	-	-	-

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	If the passed parameter is a mandatory input, output, or mandatory attribute and is a null pointer, 161001 is returned.
ACLNN_ERR_PARAM_INVALID	161002	The data type of the input or output is not within the supported range.
ACLNN_ERR_PARAM_INVALID	161002	The input or output parameters do not meet the constraints in the parameter description.

aclnnGroupNormSiluQuant

Parameters

<table style="undefined;table-layout: fixed; width: 953px"><colgroup>
  <col style="width: 173px">
  <col style="width: 112px">
  <col style="width: 668px">
  </colgroup>
      <thead>
          <tr><th>Parameter</th><th>Input/Output</th><th>Description</th></tr>
      </thead>
      <tbody>
          <tr><td>workspace</td><td>Input</td><td>Workspace memory address applied on the Device side.</td></tr>
          <tr><td>workspaceSize</td><td>Input</td><td>Workspace size applied on the Device side, obtained from the first-phase API aclnnGroupNormSiluQuantGetWorkspaceSize.</td></tr>
          <tr><td>executor</td><td>Input</td><td>Operator executor, which includes the operator computation flow.</td></tr>
          <tr><td>stream</td><td>Input</td><td>Specifies the stream that executes the task.</td></tr>
      </tbody>
  </table>

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnGroupNormSiluQuant defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_group_norm_silu_quant.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shape_size = 1;
  for (auto i : shape) {
    shape_size *= i;
  }
  return shape_size;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate for AscendCL initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to apply for memory on the device side.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

  // Copy the host-side data to the device memory by calling aclrtMemcpy.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Create an aclTensor by calling the aclCreateTensor API.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. For details, see the ACL API list.
  // Set deviceId based on the actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  // Handle based on your own needs.
  CHECK_RET(ret == 0, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
  // 2. Construct the input and output. Custom construction is required based on the API interface.
  std::vector<int64_t> selfShape = {1, 2, 4, 4};    // Main input
  std::vector<int64_t> gammaShape = {2};      // gamma parameter
  std::vector<int64_t> betaShape = {2};       // beta parameter
  std::vector<int64_t> quantScaleShape = {1}; // Quantization scaling factor.
  std::vector<int64_t> outShape = {1, 2, 4, 4};     // Quantized output.
  std::vector<int64_t> meanOutShape = {1, 2};  // Mean output [N, G].
  std::vector<int64_t> rstdOutShape = {1, 2};   // Reciprocal standard deviation output [N, G].

  void* selfDeviceAddr = nullptr;
  void* gammaDeviceAddr = nullptr;
  void* betaDeviceAddr = nullptr;
  void* quantScaleDeviceAddr = nullptr;
  void* outDeviceAddr = nullptr;
  void* meanOutDeviceAddr = nullptr;
  void* rstdOutDeviceAddr = nullptr;
  aclTensor* self = nullptr;
  aclTensor* gamma = nullptr;
  aclTensor* beta = nullptr;
  aclTensor* quantScale = nullptr;
  aclTensor* out = nullptr;
  aclTensor* meanOut = nullptr;
  aclTensor* rstdOut = nullptr;
  // Use uint16_t to store the bit pattern of BF16/FP16.
  std::vector<uint16_t> selfHostData = {
  0x3C00, 0x4000, 0x4200, 0x4400, // FP16 bit pattern of 1.0, 2.0, 3.0, 4.0
  0x4500, 0x4600, 0x4700, 0x4800, // FP16 bit pattern of 5.0, 6.0, 7.0, 8.0
  0x3C00, 0x4000, 0x4200, 0x4400, // FP16 bit pattern of 1.0, 2.0, 3.0, 4.0
  0x4500, 0x4600, 0x4700, 0x4800, // FP16 bit pattern of 5.0, 6.0, 7.0, 8.0
  0x3C00, 0x4000, 0x4200, 0x4400, // FP16 bit pattern of 1.0, 2.0, 3.0, 4.0
  0x4500, 0x4600, 0x4700, 0x4800, // FP16 bit patterns for 5.0, 6.0, 7.0, 8.0
  0x3C00, 0x4000, 0x4200, 0x4400, // FP16 bit patterns for 1.0, 2.0, 3.0, 4.0
  0x4500, 0x4600, 0x4700, 0x4800 // FP16 bit patterns for 5.0, 6.0, 7.0, 8.0
  };
  std::vector<uint16_t> gammaHostData = {0x3C00, 0x3C00}; // FP16 bit patterns for 1.0, 1.0
  std::vector<uint16_t> betaHostData = {0x0000, 0x0000}; // FP16 bit patterns for 0.0, 0.0
  std::vector<float> quantScaleHostData = {1.0};
  std::vector<int8_t> outHostData = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                                    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; // Initialized to 0, and actually filled by the operator.
  // The statistic output also uses half-precision format.
  std::vector<uint16_t> meanOutHostData = {0x0000, 0x0000}; // Initialized to 0.
  std::vector<uint16_t> rstdOutHostData = {0x0000, 0x0000}; // Initialized to 0.

  int64_t group = 2;
  double eps = 0.00001;
  bool activateSilu = true;
  // Create the self aclTensor.
  ret = CreateAclTensor(selfHostData, selfShape, &selfDeviceAddr, aclDataType::ACL_FLOAT16, &self);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the gamma aclTensor.
  ret = CreateAclTensor(gammaHostData, gammaShape, &gammaDeviceAddr, aclDataType::ACL_FLOAT16, &gamma);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the beta aclTensor.
  ret = CreateAclTensor(betaHostData, betaShape, &betaDeviceAddr, aclDataType::ACL_FLOAT16, &beta);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the quantScale aclTensor.
  ret = CreateAclTensor(quantScaleHostData, quantScaleShape, &quantScaleDeviceAddr, aclDataType::ACL_FLOAT, &quantScale);
  CHECK_RET(ret == ACL_SUCCESS, return ret);  // Create the out aclTensor.
  ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT8, &out);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the meanOut aclTensor.
  ret = CreateAclTensor(meanOutHostData, meanOutShape, &meanOutDeviceAddr, aclDataType::ACL_FLOAT16, &meanOut);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the rstdOut aclTensor.
  ret = CreateAclTensor(rstdOutHostData, rstdOutShape, &rstdOutDeviceAddr, aclDataType::ACL_FLOAT16, &rstdOut);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // 3. Call the CANN operator library API. Replace with the specific API.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  // Call the first-phase API of aclnnGroupNormSiluQuant.
  ret = aclnnGroupNormSiluQuantGetWorkspaceSize(self, gamma, beta, quantScale, group, eps, activateSilu, out, meanOut, rstdOut, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupNormSiluQuantGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Apply for device memory based on the workspaceSize calculated by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
  }
  // Call the second-phase API of aclnnGroupNormSiluQuant.
  ret = aclnnGroupNormSiluQuant(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupNormSiluQuant failed. ERROR: %d\n", ret); return ret);
  // 4. (Boilerplate) Synchronously wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
  // 5. Obtain the output value and copy the result from the device-side memory to the host side. Modify based on the specific API definition.
  auto size = GetShapeSize(outShape);
  std::vector<int8_t> outResultData(size, 0);
  ret = aclrtMemcpy(outResultData.data(), outResultData.size() * sizeof(outResultData[0]), outDeviceAddr, size * sizeof(int8_t),
                    ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("outResultData[%ld] is: %d\n", i, outResultData[i]);
  }

  // Receive the meanOut result.
  size = GetShapeSize(meanOutShape);
  std::vector<uint16_t> meanResultData(size, 0);
  ret = aclrtMemcpy(meanResultData.data(),
                    meanResultData.size() * sizeof(uint16_t), // 2 bytes
                    meanOutDeviceAddr,
                    size * sizeof(uint16_t), // 2 bytes
                    ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy meanOut from device to host failed. ERROR: %d\n", ret); return ret);

  // Receive the rstdOut result.
  size = GetShapeSize(rstdOutShape);
  std::vector<uint16_t> rstdResultData(size, 0);
  ret = aclrtMemcpy(rstdResultData.data(),
                    rstdResultData.size() * sizeof(uint16_t), // 2 bytes
                    rstdOutDeviceAddr,
                    size * sizeof(uint16_t), // 2 bytes
                    ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy rstdOut from device to host failed. ERROR: %d\n", ret); return ret);

  // Print the result (converted to float for readability).
  for (int64_t i = 0; i < meanResultData.size(); i++) {
    // Simple conversion: convert the FP16 bit pattern to float.
    __fp16 fp16_val = *reinterpret_cast<__fp16*>(&meanResultData[i]);
    float fp32_val = static_cast<float>(fp16_val);
    LOG_PRINT("meanResultData[%ld] is: %f\n", i, fp32_val);
  }

  for (int64_t i = 0; i < rstdResultData.size(); i++) {
    __fp16 fp16_val = *reinterpret_cast<__fp16*>(&rstdResultData[i]);
    float fp32_val = static_cast<float>(fp16_val);
    LOG_PRINT("rstdResultData[%ld] is: %f\n", i, fp32_val);
  }

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(self);
  aclDestroyTensor(gamma);
  aclDestroyTensor(beta);
  aclDestroyTensor(quantScale);
  aclDestroyTensor(out);
  aclDestroyTensor(meanOut);
  aclDestroyTensor(rstdOut);

  // 7. Release device resources. Modify based on the specific API definition.
  aclrtFree(selfDeviceAddr);
  aclrtFree(gammaDeviceAddr);
  aclrtFree(betaDeviceAddr);
  aclrtFree(quantScaleDeviceAddr);
  aclrtFree(outDeviceAddr);
  aclrtFree(meanOutDeviceAddr);
  aclrtFree(rstdOutDeviceAddr);

  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}