Documentation
Download
文档中心
aclnnDynamicQuantV2

aclnnDynamicQuantV2

📄 View Source Code

Applicable Products

ProductSupported
Ascend 950PR/Ascend 950DT√
Atlas A3 training products/Atlas A3 inference products√
Atlas A2 training products/Atlas A2 inference products√
Atlas 200I/500 A2 inference products×
Atlas inference products√
Atlas training products√

Function

  • Description: Performs per-token symmetric/asymmetric dynamic quantization on the input tensor. In MOE scenarios, the smoothScalesOptional of each expert is different and is distinguished based on the input groupIndexOptional.

  • Formula:

    • Symmetric quantization:

      • If smoothScalesOptional is not input:

        scaleOut=row_max(abs(x))/127scaleOut=row\_max(abs(x))/127 yOut=round(x/scaleOut)yOut=round(x/scaleOut)

      • If smoothScalesOptional is input:

        input=x⋅smoothScalesOptionalinput = x\cdot smoothScalesOptional scaleOut=row_max(abs(input))/127scaleOut=row\_max(abs(input))/127 yOut=round(input/scaleOut)yOut=round(input/scaleOut)

    • Asymmetric quantization:

      • If smoothScalesOptional is not input,

        scaleOut=(row_max(x)−row_min(x))/scale_optscaleOut=(row\_max(x) - row\_min(x))/scale\_opt offset=offset_opt−row_max(x)/scaleOutoffset=offset\_opt-row\_max(x)/scaleOut yOut=round(x/scaleOut+offset)yOut=round(x/scaleOut+offset)

      • If smoothScalesOptional is input, then

        input=x⋅smoothScalesOptionalinput = x\cdot smoothScalesOptional scaleOut=(row_max(input)−row_min(input))/scale_optscaleOut=(row\_max(input) - row\_min(input))/scale\_opt offset=offset_opt−row_max(input)/scaleOutoffset=offset\_opt-row\_max(input)/scaleOut yOut=round(input/scaleOut+offset)yOut=round(input/scaleOut+offset)

    Where row_max represents the maximum value of each row, and row_min represents the minimum value of each row. When the output yOut type is INT8, scale_opt is 255.0 and offset_opt is 127.0; when the yOut type is INT4 or INT32, scale_opt is 15.0 and offset_opt is 7.0.

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnDynamicQuantV2GetWorkspaceSize" API to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow, and then call the "aclnnDynamicQuantV2" API to perform the computation.

aclnnStatus aclnnDynamicQuantV2GetWorkspaceSize(
  const aclTensor *x,
  const aclTensor *smoothScalesOptional,
  const aclTensor *groupIndexOptional,
  int64_t          dstType,
  const aclTensor *yOut,
  const aclTensor *scaleOut,
  const aclTensor *offsetOut,
  uint64_t        *workspaceSize,
  aclOpExecutor  **executor)
aclnnStatus aclnnDynamicQuantV2(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnDynamicQuantV2GetWorkspaceSize

  • Parameters:

    Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor
    x (aclTensor*) Input Operator input tensor. Corresponds to `x` in the formula.
    • Supports empty tensors.
    		 FLOAT16, BFLOAT16 ND Greater than one dimension. √
    smoothScalesOptional (aclTensor*) Input SmoothScales of the operator input. Corresponds to `smoothScalesOptional` in the formula description.
    • Supports empty tensors.
    • The data type must be consistent with x.
    • When groupIndexOptional is absent, the shape dimension is the same as the last dimension of x. When groupIndexOptional is present, the shape is two-dimensional, where the first dimension is the number of experts (not exceeding 1024) and the second dimension is the last dimension of x.
    		 FLOAT16, BFLOAT16 ND 2 √
    groupIndexOptional (aclTensor*) Input Operator input groupIndex.
    • Supports empty tensor.
    • Has only one dimension.
    		 FLOAT16, BFLOAT16 ND 1 √
    dstType (int64_t) input Enumeration value corresponding to the output y type.
    • If the output y type is INT8, the value is 2; if the y type is INT4, the value is 29; if the y type is INT32, the value is 3.
    		 - - - -
    yOut (aclTensor*) Output Quantized output tensor. Corresponds to `yOut` in the corresponding formula.
    • Supports empty tensors.
    • When the type is INT32, the last dimension of the shape is 1/8 of the last dimension of x, and the other dimensions are consistent with x. For other types, the shape is consistent with x.
    		 INT4, INT8, INT32, FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8 ND More than one dimension. -
    scaleOut (aclTensor*) Output The scale used for quantization. Corresponds to `scaleOut` in the formula.
    • Supports empty tensor.
    • The shape dimension is the shape of x with the last dimension removed.
    		 FLOAT32 ND Greater than 0 dimensions. -
    offsetOut (aclTensor*) Output Offset used for quantization. Corresponds to `offset` in the formula.
    • Empty tensor is supported.
    • The shape dimensions are the shape of x with the last dimension removed.
    		 FLOAT32 ND Greater than 0 dimensions -
    workspaceSize (uint64_t*) Output Returns the workspace size to be allocated on the Device side. - - - - -
    executor (aclOpExecutor**) Output Returns the operator executor, which contains the operator computation flow. - - - - -

    • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

      The data type of the output yOut only supports INT4, INT8, and INT32.

    • Atlas inference products, Atlas training products:

      • The input parameter x only supports FLOAT16.

      • The input parameters smoothScalesOptional and groupIndexOptional are reserved parameters and do not participate in computation in the current version.

      • The input parameter dstType only supports being set to 2.

      • The output parameter yOut only supports INT8.

  • Return value

    aclnnStatus: return code. For details, see aclnn Return Code.

    The first-phase API performs input parameter validation and returns an error in the following scenarios:

    Return Code Error Code Description
    ACLNN_ERR_PARAM_NULLPTR 161001 The input x or yOut parameter is a null pointer.
    ACLNN_ERR_PARAM_INVALID 161002 The data type, data format, or dimension of the parameter is not within the supported range.
    When the data type of yOut is INT4, the last dimension of x or yOut is not an even number.
    When the data type of yOut is INT32, the last dimension of x is not divisible by 8.
    ACLNN_ERR_INNER_CREATE_EXECUTOR 561001 Failed to create an internal aclOpExecutor.

aclnnDynamicQuantV2

  • Parameters:

    Parameter Input/Output Description
    workspace Input Address of the workspace memory applied for on the Device side.
    workspaceSize Input Size of the workspace applied for on the Device side, obtained from the first-phase API aclnnDynamicQuantV2GetWorkspaceSize.
    executor Input Operator executor that contains the operator computation flow.
    stream Input Specifies the stream for task execution.

  • Return value

    aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

  • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT:

  • When the data type of yOut is INT4, the last dimension of x and yOut must be divisible by 2.

  • When the data type of yOut is INT32, the last dimension of x must be divisible by 8.

  • Atlas inference products and Atlas training products: The last dimension of input x must be divisible by 32. Currently, only symmetric quantization is supported, and the BFLOAT16 data type is not supported.

  • Deterministic Computation:

    • aclnnDynamicQuantV2 defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_dynamic_quant_v2.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

void PrintOutResult(std::vector<int64_t> &shape, void** deviceAddr) {
    auto size = GetShapeSize(shape);
    std::vector<float> resultData(size, 0);
    auto ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]),
                           *deviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("mean result[%ld] is: %f\n", i, resultData[i]);
    }
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(float);
    // Call aclrtMalloc to allocate memory on the device side.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy data from the host side to the device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

int main() {
    // 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
    // Enter the deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Customize the construction based on the API.
    int rowNum = 4;
    int rowLen = 32;
    int groupNum = 4;
    std::vector<int64_t> xShape = {4, 32};
    std::vector<int64_t> smoothShape = {groupNum, rowLen};
    std::vector<int64_t> groupShape = {groupNum};
    std::vector<int64_t> yShape = {4, 32};
    std::vector<int64_t> scaleShape = {4};
    std::vector<int64_t> offsetShape = {4};

    void* xDeviceAddr = nullptr;
    void* smoothDeviceAddr = nullptr;
    void* groupDeviceAddr = nullptr;
    void* yDeviceAddr = nullptr;
    void* scaleDeviceAddr = nullptr;
    void* offsetDeviceAddr = nullptr;

    aclTensor* x = nullptr;
    aclTensor* smooth = nullptr;
    aclTensor* group = nullptr;
    aclTensor* y = nullptr;
    aclTensor* scale = nullptr;
    aclTensor* offset = nullptr;

    std::vector<float> xHostData;
    std::vector<float> smoothHostData;
    std::vector<int32_t> groupHostData = {2, rowNum};
    std::vector<int8_t> yHostData;
    std::vector<float> scaleHostData;
    std::vector<float> offsetHostData;
    for (int i = 0; i < rowNum; ++i) {
        for (int j = 0; j < rowLen; ++j) {
            float value1 = i * rowLen + j;
            xHostData.push_back(aclFloatToFloat16(value1));
            yHostData.push_back(0);
        }
        scaleHostData.push_back(0);
        offsetHostData.push_back(0);
    }

    for (int m = 0; m < groupNum; ++m) {
        for (int n = 0; n < rowLen; ++n) {
            float value2 = m * rowLen + n;
            smoothHostData.push_back(aclFloatToFloat16(value2));
        }
    }

    // Create the x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create smooth aclTensor.
    ret = CreateAclTensor(smoothHostData, smoothShape, &smoothDeviceAddr, aclDataType::ACL_FLOAT16, &smooth);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create group aclTensor.
    ret = CreateAclTensor(groupHostData, groupShape, &groupDeviceAddr, aclDataType::ACL_INT32, &group);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create y aclTensor.
    ret = CreateAclTensor(yHostData, yShape, &yDeviceAddr, aclDataType::ACL_INT8, &y);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create scale aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create offset aclTensor.
    ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // 3. Call the CANN operator library API. Replace it with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;

    // Call the first-phase API of aclnnDynamicQuantV2.
    ret = aclnnDynamicQuantV2GetWorkspaceSize(x, smooth, group, 2,  y, scale, offset, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnDynamicQuantV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);

    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }

    // Call the second-phase API of aclnnDynamicQuantV2.
    ret = aclnnDynamicQuantV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnDynamicQuantV2 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Wait synchronously for the task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device side memory to the host side. Modify based on the specific API definition.
    PrintOutResult(yShape, &yDeviceAddr);

    // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyTensor(smooth);
    aclDestroyTensor(y);
    aclDestroyTensor(scale);
    aclDestroyTensor(offset);

    // 7. Release device resources.
    aclrtFree(xDeviceAddr);
    aclrtFree(smoothDeviceAddr);
    aclrtFree(yDeviceAddr);
    aclrtFree(scaleDeviceAddr);
    aclrtFree(offsetDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();

    return 0;
}
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details