aclnnGeluQuant

Applicable Products

Function

• Description: Fuses GeluV2 with DynamicQuant/AscendQuantV2. It performs Gelu activation on the input data self, and then quantizes the activation result, outputting the quantized result.

• Formula:

1. First, compute gelu to obtain geluOut.

• approximate = tanh

• approximate = none

2. Then perform quantization on geluOut.

• quant_mode = static

• quant_mode = dynamic

  $$geluOut = geluOut * inputScaleOptional$$ $$Max = max(abs(geluOut))$$ $$outScaleOptional = Max/maxValue$$ $$y = round\_to\_dst\_type(geluOut / outScaleOptional, round\_mode)$$

• maxValue: The maximum value of the corresponding data type.

  DataType	maxValue
  INT8	127
  FLOAT8_E4M3FN	448
  FLOAT8_E5M2	57344
  HIFLOAT8	32768

Function Prototype

Each operator is divided into a two-phase API. The "aclnnGeluQuantGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnGeluQuant" API is called to perform the computation.

aclnnStatus aclnnGeluQuantGetWorkspaceSize(
    const aclTensor* self,
    const aclTensor* inputScaleOptional,
    const aclTensor* inputOffsetOptional,
    const char*      approximate,
    const char*      quantMode,
    const char*      roundMode,
    int64_t          dstType,
    const aclTensor* y,
    const aclTensor* outScaleOptional,
    uint64_t*        workspaceSize,
    aclOpExecutor**  executor)

aclnnStatus aclnnGeluQuant(
    void*            workspace,
    uint64_t         workspaceSize,
    aclOpExecutor*   executor,
    aclrtStream      stream)

aclnnGeluQuantGetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-Contiguous Tensor
  self (aclTensor*)	Input	Input self in the formula.	Does not support null tensor. When quantMode is "dynamic", shape supports 2–8 dimensions. When quantMode is "static", shape supports 1–8 dimensions.	FLOAT16, BFLOAT16, FLOAT	ND	1-8	√
  inputScaleOptional（aclTensor*）	Input	Input of the operator, inputScaleOptional in the formula.	Does not support null tensors. Shape only supports 1D, and its size can only be the last axis dimension of self or 1. Required input when quantMode is static, and optional input when quantMode is dynamic.	FLOAT16, BFLOAT16, FLOAT	ND	1	√
  inputOffsetOptional (aclTensor*)	Input	Optional input of the operator, representing inputOffsetOptional in the formula.	Does not support null Tensor. The shape only supports 1D and must be consistent with the dtype and shape of inputScaleOptional. When quantMode is dynamic and inputScaleOptional is not provided, offset is not provided.	FLOAT16, BFLOAT16, FLOAT	ND	1	√
  approximate (char*)	Input	approximate in the formula, the mode of the GELU activation function.	approximate only supports {"none", "tanh"}.	-	-	-	-
  quantMode（char*）	Input	quantMode in the formula, which specifies the quantization mode.	quantMode only supports {"static", "dynamic"}, corresponding to static quantization and dynamic quantization respectively.	-	-	-	-
  roundMode (char*)	Input	Approximate in the formula, the mode of the GELU activation function.	Supports {"rint", "round", "hybrid"} modes. When dstType is 2/35/36, corresponding to the INT8/FLOAT8_E4M3FN/FLOAT8_E5M2 data types, only {"rint"} is supported. When dstType is 34, corresponding to the HIFLOAT8 data type, {"round", "hybrid"} are supported.	-	-	-	-
  dstType（int64_t）	Input	dst_type in the formula.	Specifies the type of output y after data conversion. The input range is {2, 34, 35, 36}, corresponding to the output y data types {2: INT8, 34: HIFLOAT8, 35: FLOAT8_E5M2, 36: FLOAT8_E4M3FN} respectively.	-	-	-	-
  y (aclTensor*)	Output	Corresponding result after activation output and quantization, i.e., y in the formula.	Does not support null tensor. The data type must correspond to dstType. Must have the same shape size as self.	FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, INT8	ND	1-8	√
  outScaleOptional (aclTensor*)	Output	Quantization scale for dynamic quantization, outScaleOptional in the formula.	Does not support null tensor. The shape has one dimension fewer than self. The dimension sizes are the same as those of self except for the last dimension. When quantMode is static, the outScaleOptional output should be a null pointer.	FLOAT	ND	1-7	√
  workspaceSize (uint64_t*)	Output	Returns the workspace size to be applied on the Device side.	-	-	-	-	-
  executor (aclOpExecutor**)	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

• Return value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

aclnnGeluQuant

• Parameters:

  Parameter	Input/Output	Description
  workspace	Input	Memory address of the workspace applied on the Device side.
  workspaceSize	Input	Size of the workspace applied on the Device side, obtained from the first-phase API aclnnGeluQuantGetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic computation:

  • aclnnGeluQuant defaults to a deterministic implementation.

• The data type of inputScaleOptional is the same as that of self, or a higher-precision type is used when the types differ.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_gelu_quant.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // boilerplate, resource initialization
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType, aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to the device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int aclnnGeluQuantTest(int32_t deviceId, aclrtStream& stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Custom construction is required based on the API.
    std::vector<int64_t> xShape = {4, 2};
    std::vector<int64_t> inputScaleShape = {1};
    std::vector<int64_t> inputOffsetShape = {1};
    std::vector<int64_t> yOutShape = {4, 2};
    std::vector<int64_t> emptyShape = {4, 0};
    void* xDeviceAddr = nullptr;
    void* inputScaleDeviceAddr = nullptr;
    void* inputOffsetDeviceAddr = nullptr;
    void* yOutDeviceAddr = nullptr;
    void* outscaleOutDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclTensor* inputScale = nullptr;
    aclTensor* inputOffset = nullptr;
    aclTensor* yOut = nullptr;
    aclTensor* outScale = nullptr;
    std::vector<float> xHostData = {1.3, 2.5, 6.7, -4, -1.4, -1.6, -8, -16.9};
    std::vector<float> inputScaleHostData = {1};
    std::vector<uint8_t> yOutHostData = {1, 2, 7, 0, 0, 0, 0, 0};
    const char* approximate = "tanh";
    const char* quantMode = "static";
    const char* roundMode = "rint";
    int64_t dstType = 2;
    // Create the x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> xTensorPtr(x, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> xDeviceAddrPtr(xDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the inputScale aclTensor.
    ret = CreateAclTensor(inputScaleHostData, inputScaleShape, &inputScaleDeviceAddr, aclDataType::ACL_FLOAT, &inputScale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> inputScaleTensorPtr(inputScale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> inputScaleDeviceAddrPtr(inputScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the yOut aclTensor.
    ret = CreateAclTensor(yOutHostData, yOutShape, &yOutDeviceAddr, aclDataType::ACL_INT8, &yOut);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> yOutTensorPtr(yOut, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> yOutDeviceAddrPtr(yOutDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;

    // Call the first-phase API of aclnnGeluQuant.
    ret = aclnnGeluQuantGetWorkspaceSize(x, inputScale, inputOffset, approximate, quantMode, roundMode, dstType, yOut, outScale, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGeluQuantGetWorkspaceSize failed. ERROR: %d\n", ret);
            return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnGeluQuant.
    ret = aclnnGeluQuant(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGeluQuant failed. ERROR: %d\n", ret); return ret);

    //(Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // Obtain the output value and copy the result from the device memory to the host memory. Modify based on the specific API definition.
    auto size = GetShapeSize(yOutShape);
    std::vector<int8_t> yOutData(
        size, 0);
    ret = aclrtMemcpy(yOutData.data(), yOutData.size() * sizeof(yOutData[0]), yOutDeviceAddr,
                    size * sizeof(yOutData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy yOut from device to host failed. ERROR: %d\n", ret);
            return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("y[%ld] is: %d\n", i, yOutData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnGeluQuantTest(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGeluQuantTest failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}