aclnnAdaLayerNormQuant

📄 View Source Code

Applicable Products

Function

• Description: The AdaLayerNormQuant operator fuses AdaLayerNorm and downstream quantization (currently only DynamicQuant is supported). This operator is primarily used to perform quantization on adaptive layer normalization, i.e., normalizing the input data and quantizing it into low-precision integers to improve computational efficiency and reduce memory usage.

• Formula:

  1. First, perform LayerNorm normalization on the input x:
  $$LayerNorm(x) = {{x-E(x)}\over\sqrt {Var(x)+epsilon}} * weightOptional + biasOptional$$
  2. Then adjust the normalization result using the adaptive parameters scale and shift:
  $$y = LayerNorm(x) * (1 + scale) + shift$$
  3. If smoothScalesOptional is not null:
  $$y = y \cdot smoothScalesOptional$$
  4. Then compute the maximum absolute value of y and divide it by $(FP8\_MAX / HIF8\_MAX / INT8\_MAX)$ to obtain the quantization scale factor for quantizing to FLOAT8/HIF8/INT8 format:
  $$quantScale = row\_max(abs(y)) / (FP8\_MAX / HIF8\_MAX / INT8\_MAX)$$
  5. Finally, divide y by the quantization scale factor and round to obtain the quantized output:
  $$out = round(y / quantScale)$$

  Where E(x) represents the mean of the input, Var(x) represents the variance of the input, and row_max represents the maximum value of each row.

Function Prototype

Each operator is divided into a two-phase API. The "aclnnAdaLayerNormQuantGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnAdaLayerNormQuant" API is called to perform the computation.

aclnnStatus aclnnAdaLayerNormQuantGetWorkspaceSize(
  const aclTensor* x,
  const aclTensor* scale,
  const aclTensor* shift,
  const aclTensor* weightOptional,
  const aclTensor* biasOptional,
  const aclTensor* smoothScalesOptional,
  double           epsilon,
  const char*      quantMode,
  aclTensor*       out,
  aclTensor*       quantScale,
  aclTensor*       quantOffsetOptional,
  uint64_t*        workspaceSize,
  aclOpExecutor**  executor)

aclnnStatus aclnnAdaLayerNormQuant(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnAdaLayerNormQuantGetWorkspaceSize

• Parameters

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (shape)	Non-contiguous Tensor
  x (aclTensor*)	Input	Represents the input data to be processed. Corresponds to `x` in the formula.	Empty tensors are not supported. The shape is [B, S, H], where B supports 0 to 6 dimensions.	FLOAT16, BFLOAT16	ND	2-8	√
  scale (aclTensor*)	Input	Represents the adaptive scale parameter. Corresponds to `scale` in the formula.	Empty tensors are not supported. The data type is the same as that of the input `x`. The shape is [B, H] or [B, 1, H], where B supports 0 to 6 dimensions, the number and size of dimensions are consistent with B in `x`, and H is consistent with the H dimension in `x`.	FLOAT16, BFLOAT16	ND	1-8	√
  shift (aclTensor*)	Input	Represents the adaptive offset parameter. Corresponds to `shift` in the formula.	Empty tensors are not supported. The data type is the same as that of the input x. The shape is [B, H] or [B, 1, H], where B supports 0 to 6 dimensions. The number of dimensions and sizes of B must be consistent with those of B in x, and H must be consistent with the H dimension in x.	FLOAT16, BFLOAT16	ND	1-8	√
  weightOptional (aclTensor*)	Input	Optional input parameter that represents the normalization scale parameter. Corresponds to `weightOptional` in the formula.	Empty tensors are not supported. The data type must be the same as that of the input parameter `x`. The shape is [H], where H must be consistent with the H dimension in `x`.	FLOAT16, BFLOAT16	ND	1	√
  biasOptional (aclTensor*)	Input	Optional input parameter, representing the normalization bias parameter. Corresponds to `biasOptional` in the formula.	Empty tensors are not supported. The data type must be the same as that of the input parameter `x`. The shape is [H], where H must be consistent with the H dimension in `x`.	FLOAT16, BFLOAT16	ND	1	√
  smoothScalesOptional (aclTensor*)	Input	Optional input parameter, representing the smoothing weight for quantization. Corresponds to `smoothScalesOptional` in the corresponding formula.	Does not support empty tensors. The data type is the same as that of the input `x`. The shape is [H], where H is consistent with the H dimension in `x`.	FLOAT16, BFLOAT16	ND	1	√
  epsilon (double)	Input	Represents the value added to the denominator to ensure numerical stability. Corresponds to `epsilon` in the formula.	-	-	-	-	-
  quantMode (char*)	Input	Indicates the quantization mode.	Currently, only "dynamic" is supported.	-	-	-	-
  out (aclTensor*)	Output	Represents the quantized output tensor. Corresponds to `out` in the formula.	Does not support empty tensors. The shape must be consistent with the shape of the input `x`.	INT8, FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8	ND	2-8	√
  quantScale (aclTensor*)	Output	Represents the quantization coefficient. Corresponds to `quantScale` in the formula.	Does not support empty tensors. Shape is [B, S], where B supports 0 to 6 dimensions, with the number of dimensions and size consistent with B in `x`, and S consistent with the S dimension in `x`.	FLOAT32	ND	1-7	√
  quantOffsetOptional (aclTensor*)	Output	Optional output, representing the offset used in asymmetric quantization.	Empty tensors are not supported. The shape must be consistent with the shape of quantScale. Not supported in the current version. Pass nullptr.	FLOAT16, BFLOAT16	ND	1-7	√
  workspaceSize (uint64_t*)	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
  executor（aclOpExecutor**）	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

  • Atlas A3 training products/Atlas A3 inference products, Atlas A2 training products/Atlas A2 inference products: The data type of the output out supports only INT8.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

aclnnAdaLayerNormQuant

• Parameters

  Parameter	Input/Output	Description
  workspace	Input	Address of the workspace memory applied for on the Device side.
  workspaceSize	Input	Size of the workspace applied for on the Device side, obtained through the first-phase API aclnnAdaLayerNormQuantGetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic computation:

  • aclnnAdaLayerNormQuant defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_ada_layer_norm_quant.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shape_size = 1;
    for (auto i : shape) {
        shape_size *= i;
    }
    return shape_size;
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Copy host-side data to device-side memory by calling aclrtMemcpy.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Create an aclTensor by calling the aclCreateTensor API.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

int main() {
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check result as needed.
    CHECK_RET(ret == 0, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
    // 2. Construct inputs and outputs as needed based on the API.
    std::vector<int64_t> xShape = {2, 4, 8};
    std::vector<int64_t> scaleShape = {2, 8};
    std::vector<int64_t> shiftShape = {2, 8};
    std::vector<int64_t> weightShape = {8};
    std::vector<int64_t> biasShape = {8};
    std::vector<int64_t> smoothScalesShape = {8};
    std::vector<int64_t> outShape = {2, 4, 8};
    std::vector<int64_t> quantScaleShape = {2, 4};
    double epsilon = 1e-5;
    const char* quantMode = "dynamic";
    void* xDeviceAddr = nullptr;
    void* scaleDeviceAddr = nullptr;
    void* shiftDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* smoothScalesDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* quantScaleDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclTensor* scale = nullptr;
    aclTensor* shift = nullptr;
    aclTensor* weight = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* smoothScales = nullptr;
    aclTensor* out = nullptr;
    aclTensor* quantScale = nullptr;
    std::vector<short> xHostData(2 * 4 * 8, 1);
    std::vector<short> scaleHostData(2 * 8, 1);
    std::vector<short> shiftHostData(2 * 8, 1);
    std::vector<short> weightHostData(8, 1);
    std::vector<short> biasHostData(8, 1);
    std::vector<short> smoothScalesHostData(8, 1);
    std::vector<int8_t> outHostData(2 * 4 * 8, 0);
    std::vector<float> quantScaleHostData(2 * 4, 0);
    // Create the x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the scale aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT16, &scale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the shift aclTensor.
    ret = CreateAclTensor(shiftHostData, shiftShape, &shiftDeviceAddr, aclDataType::ACL_FLOAT16, &shift);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the weight aclTensor.
    ret = CreateAclTensor(weightHostData, weightShape, &weightDeviceAddr, aclDataType::ACL_FLOAT16, &weight);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_FLOAT16, &bias);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the smoothScales aclTensor.
    ret = CreateAclTensor(smoothScalesHostData, smoothScalesShape, &smoothScalesDeviceAddr, aclDataType::ACL_FLOAT16, &smoothScales);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT8, &out);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the quantScale aclTensor.
    ret = CreateAclTensor(quantScaleHostData, quantScaleShape, &quantScaleDeviceAddr, aclDataType::ACL_FLOAT, &quantScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // 3. Call the CANN operator library API. Replace with the specific API.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnAdaLayerNormQuant.
    ret = aclnnAdaLayerNormQuantGetWorkspaceSize(x, scale, shift, weight, bias, smoothScales, epsilon, quantMode, out, quantScale, nullptr, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAdaLayerNormQuantGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Allocate device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
    }
    // Call the second-phase API of aclnnAdaLayerNormQuant.
    ret = aclnnAdaLayerNormQuant(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAdaLayerNormQuant failed. ERROR: %d\n", ret); return ret);
    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
    // 5. Obtain the output value and copy the result from the device memory to the host. Modify based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<int8_t> resultData(size, 0);
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(int8_t),
                      ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
    }

    // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyTensor(scale);
    aclDestroyTensor(shift);
    aclDestroyTensor(weight);
    aclDestroyTensor(bias);
    aclDestroyTensor(smoothScales);
    aclDestroyTensor(out);
    aclDestroyTensor(quantScale);

    // 7. Release device resources. Modify based on the specific API definition.
    aclrtFree(xDeviceAddr);
    aclrtFree(scaleDeviceAddr);
    aclrtFree(shiftDeviceAddr);
    aclrtFree(weightDeviceAddr);
    aclrtFree(biasDeviceAddr);
    aclrtFree(smoothScalesDeviceAddr);
    aclrtFree(outDeviceAddr);
    aclrtFree(quantScaleDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}