aclnnGroupQuant

📄 View Source Code

Applicable Products

Function

• Description: Performs group quantization on the input x.

• Formula:

  $$y = round((x * scale) + offsetOptional)$$

Function Prototype

Each operator is divided into a two-phase API. The "aclnnGroupQuantGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnGroupQuant" API is called to perform the computation.

aclnnStatus aclnnGroupQuantGetWorkspaceSize(
  const aclTensor *x,
  const aclTensor *scale,
  const aclTensor *groupIndex,
  const aclTensor *offsetOptional,
  int32_t          dstType,
  aclTensor       *y,
  uint64_t        *workspaceSize,
  aclOpExecutor  **executor)

aclnnStatus aclnnGroupQuant(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnGroupQuantGetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  x (aclTensor*)	Input	Indicates the input to be quantized, corresponding to `x` in the formula.	Supports empty tensors. If `dstType` is 3 (INT32), the last dimension of the shape must be divisible by 8. If `dstType` is 29 (INT4), the last dimension of the shape must be divisible by 2.	FLOAT32, FLOAT16, BFLOAT16	ND	2	√
  scale (aclTensor*)	Input	Indicates the scale value in quantization, corresponding to `scale` in the formula.	Supports empty tensor. The second dimension of `scale` is equal to the second dimension of x.	FLOAT32, FLOAT16, BFLOAT16	ND	2	√
  groupIndex (aclTensor*)	Input	Indicates the groupIndex value in quantization.	Supports empty tensor. The dimension equals the first dimension of scale.	INT32, INT64	ND	1	√
  offsetOptional (aclTensor*)	Input	Optional parameter that indicates the offset value in quantization, corresponding to `offsetOptional` in the formula.	Supports empty tensors. The data type is consistent with `scale`. `offsetOptional` is a single value.	FLOAT32, FLOAT16, BFLOAT16	ND	1	√
  dstType (int32_t)	Input	Specifies the output data type.	Supported values are 2, 3, and 29, indicating INT8, INT32, and INT4, respectively.	-	-	-	-
  y (aclTensor*)	Output	Indicates the output after group quantization, corresponding to `y` in the corresponding formula.	Supports empty tensors. When the type is INT32, the last dimension of the shape is 1/8 of the last dimension of `x`, and the other dimensions are consistent with x. For other types, the shape is consistent with `x`.	INT8, INT32, INT4	ND	2	√
  workspaceSize（uint64_t*）	Output	Returns the workspace size to be applied on the Device side.	-	-	-	-	-
  executor（aclOpExecutor**）	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Code	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	Indicates that x, scale, groupIndex, or y is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	The data type or data format of x, scale, groupIndex, offsetOptional, or y is not within the supported range.
  		The data types of scale and offsetOptional are inconsistent.
  		The shapes of x, scale, groupIndex, offsetOptional, and y do not meet the restriction requirements.
  		When the data type of y is INT4, the size of the last axis of x is not an even number.
  		When the data type of y is INT32, the last axis of x's shape is not 8 times the size of the last axis of y's shape, or the sizes of the non-last axes of x and y are inconsistent.

aclnnGroupQuant

• Parameters:

  Parameter	Input/Output	Description
  workspace	Input	Address of the workspace memory applied on the Device side.
  workspaceSize	Input	Size of the workspace applied on the Device side, obtained by the first-phase API aclnnGroupQuantGetWorkspaceSize.
  executor	Input	Operator executor, which contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• The input groupIndex must be a non-decreasing sequence, with the minimum value not less than 0 and the maximum value equal to the size of the 0th dimension of the input x shape. When the constraint is not met, the groupIndex parameter is not validated.

• The 0th dimension size of the input scale does not support being 0.

• The shape of the input offsetOptional currently supports only [1, ] or [].

• Deterministic computation:

  • aclnnGroupQuant defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_group_quant.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shape_size = 1;
    for (auto i : shape) {
        shape_size *= i;
    }
    return shape_size;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate: initialize resources.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Call aclrtMemcpy to copy host-side data to device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check based on your requirements.
    CHECK_RET(ret == 0, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
    // 2. Construct the input and output. Customize the construction based on the API.
    std::vector<int64_t> selfShape = {4, 2};
    std::vector<int64_t> scaleShape = {3, 2};
    std::vector<int64_t> groupIndexShape = {3};
    std::vector<int64_t> offsetShape = {1};
    std::vector<int64_t> outShape = {4, 2};
    void* selfDeviceAddr = nullptr;
    void* scaleDeviceAddr = nullptr;
    void* groupIndexDeviceAddr = nullptr;
    void* offsetDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    aclTensor* self = nullptr;
    aclTensor* scale = nullptr;
    aclTensor* groupIndex = nullptr;
    aclTensor* offset = nullptr;
    aclTensor* out = nullptr;
    std::vector<float> selfHostData = {0, 1, 2, 3, 4, 5, 6, 7};
    std::vector<float> scaleHostData = {1, 2, 1, 2, 1, 2};
    std::vector<int32_t> groupIndexHostData = {1, 2, 4};
    std::vector<float> offsetHostData = {2};
    std::vector<int8_t> outHostData = {0, 0, 0, 0, 0, 0, 0, 0};
    // Create self aclTensor.
    ret = CreateAclTensor(selfHostData, selfShape, &selfDeviceAddr, aclDataType::ACL_FLOAT, &self);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create scale aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create groupIndex aclTensor.
    ret = CreateAclTensor(
        groupIndexHostData, groupIndexShape, &groupIndexDeviceAddr, aclDataType::ACL_INT32, &groupIndex);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create offset aclTensor.
    ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT8, &out);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    const int32_t dstType = 2;

    // 3. Call the CANN operator library API. Replace it with the specific API.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnGroupQuant.
    ret = aclnnGroupQuantGetWorkspaceSize(self, scale, groupIndex, offset, dstType, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupQuantGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
    }
    // Call the second-phase API of aclnnGroupQuant.
    ret = aclnnGroupQuant(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupQuant failed. ERROR: %d\n", ret); return ret);
    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
    // 5. Obtain the output value, copy the result from the device memory to the host memory, modify based on the specific API definition, and view the data in resultData.
    auto size = GetShapeSize(outShape);
    std::vector<int8_t> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(int8_t),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
    }

    // 6. Release aclTensor. Modify based on the specific API definition.
    aclDestroyTensor(self);
    aclDestroyTensor(scale);
    aclDestroyTensor(groupIndex);
    aclDestroyTensor(offset);
    aclDestroyTensor(out);

    // 7. Release device resources. Modify based on the specific API definition.
    aclrtFree(selfDeviceAddr);
    aclrtFree(scaleDeviceAddr);
    aclrtFree(groupIndexDeviceAddr);
    aclrtFree(offsetDeviceAddr);
    aclrtFree(outDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}