aclnnQuantizedBatchNorm

Supported Products

Product	Supported
Atlas A3 training series products/Atlas A3 inference series products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference series products	×
Atlas training series products	×

Function

Description:

Dequantizes the input Tensor, normalizes the Tensor based on the input weight, bias, and epsilon, and then quantizes the Tensor based on the outputScale and outputZeroPoint.
Formula:
1. Dequantization:
$x' = (x - inputZeroPoint) * inputScale$
1. Normalization:
$y =\frac{x' - mean}{\sqrt{var + epsilon}} * weight + bias$
1. Quantization:
$output = round(\frac{y}{outputScale} + outputZeroPoint)$

Prototype

Each operator has two-phase API calls. First, aclnnQuantizedBatchNormGetWorkspaceSize is called to obtain the input parameters and compute the required workspace size based on the process. Then, aclnnQuantizedBatchNorm is called to perform computation.

aclnnStatus aclnnQuantizedBatchNormGetWorkspaceSize(
  const aclTensor* input,
  const aclTensor* mean,
  const aclTensor* var,
  const aclScalar* inputScale,
  const aclScalar* inputZeroPoint,
  const aclScalar* outputScale,
  const aclScalar* outputZeroPoint,
  aclTensor*       weight,
  aclTensor*       bias,
  float            epsilon,
  aclTensor*       output,
  uint64_t        *workspaceSize,
  aclOpExecutor  **executor)

aclnnStatus aclnnQuantizedBatchNorm(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnQuantizedBatchNormGetWorkspaceSize

Parameters:

Name	Input/Output	Description	Usage Notes	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
input	Input	Quantized data of the model input, corresponding to `x` in the formula.	Empty tensors are not supported.	INT8, UINT8, INT32	NCHW	4	√
mean	Input	Mean value of the model input data, corresponding to `mean` in the formula.	Empty tensors are not supported. The shape length is the same as the length of the channel axis of the input parameter input.	FLOAT32	ND	1	√
var	Input	Variance of the model input data, corresponding to `var` in the formula.	Empty tensors are not supported. The shape length is the same as the length of the channel axis of the input parameter `input`.	FLOAT32	ND	1	√
inputScale	Input	Scaling coefficient of the model input data, corresponding to `inputScale` in the formula.	-	FLOAT32	ND	-	-
inputZeroPoint	Input	Bias of the model input data, corresponding to `inputZeroPoint` in the formula.	The passed value cannot go beyond the value range for the data type corresponding to input. For example, the value range of the INT8-type data is [-128, 127].	INT32	ND	-	-
outputScale	Input	Scaling coefficient of the model output data, corresponding to `outputScale` in the formula.	-	FLOAT32	ND	-	-
outputZeroPoint	Input	Bias of the model output data, corresponding to `outputZeroPoint` in the formula.	The passed value cannot go beyond the value range for the data type corresponding to input. For example, the value range of the INT8-type data is [-128, 127].	INT32	ND	-	-
weight	Input	Normalized weight (optional), corresponding to `weight` in the formula.	Empty tensors are not supported. The default value is 1. The shape length is the same as the length of the channel axis of the input parameter input.	FLOAT32	ND	1	√
bias	Input	Normalized bias (optional), corresponding to `bias` in the formula.	Empty tensors are not supported. The default value is 0. The shape length is the same as the length of the channel axis of the input parameter input.	FLOAT32	ND	1	√
epsilon	Input	Value to be added to the variance to avoid division by zero, corresponding to `epsilon` in the formula.	-	FLOAT32	-	-	-
output	Output	Quantized data of the model output, corresponding to `output` in the formula.	Empty tensors are not supported. The shape, data format, and data type are the same as those of `input`.	INT8, UINT8, INT32	NCHW	4	√
workspaceSize	Output	Size of the workspace to be allocated on the device.	-	-	-	-	-
executor	Output	Operator executor, containing the operator computation process.	-	-	-	-	-

Returns:

aclnnStatus: status code. For details, see aclnn Return Codes.

The first-phase API implements input parameter verification. The following errors may be thrown:

Return	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The passed input or output is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The data type is not supported, or the parameter does not meet the preceding constraints.

aclnnQuantizedBatchNorm

Parameters:

Name	Input/Output	Description
workspace	Input	Address of the workspace to be allocated on the device.
workspaceSize	Input	Size of the workspace to be allocated on the device, which is obtained by the first-phase API aclnnQuantizedBatchNormGetWorkspaceSize.
executor	Input	Operator executor, containing the operator computation process.
stream	Input	Stream for executing the task.

Returns:

aclnnStatus: status code. For details, see aclnn Return Codes.

Constraints

Deterministic compute:
- aclnnQuantizedBatchNorm defaults to a deterministic implementation.

Example

The following example is for reference only. For details, see Compilation and Running Sample.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_quantized_batch_norm.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // (Fixed writing) Initialize resources.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate memory on the device.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Call aclrtMemcpy to copy the data on the host to the memory on the device.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of the contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

template <typename T>
int CreateAclTensorNCHW(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate memory on the device.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Call aclrtMemcpy to copy the data on the host to the memory on the device.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of the contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_NCHW, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

int main()
{
    // 1. (Fixed writing) Initialize the device and stream. For details, see the ACL API manual.
    // Set the device ID in use.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check as required.
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output based on the API definition.
    std::vector<int64_t> selfShape = {1, 2, 1, 4};
    std::vector<int64_t> meanShape = {2};
    std::vector<int64_t> varShape = {2};
    std::vector<int64_t> weightShape = {2};
    std::vector<int64_t> biasShape = {2};
    std::vector<int64_t> outShape = {1, 2, 1, 4};
    void* selfDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* rMeanDeviceAddr = nullptr;
    void* rVarDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* meanDeviceAddr = nullptr;
    void* varDeviceAddr = nullptr;
    aclTensor* self = nullptr;
    aclTensor* weight = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    aclTensor* mean = nullptr;
    aclTensor* var = nullptr;
    std::vector<int32_t> selfHostData = {0, 1, 2, 3, 4, 5, 6, 7};
    std::vector<float> weightHostData = {1, 1};
    std::vector<float> biasHostData = {0, 0};
    std::vector<float> meanHostData = {0, 0};
    std::vector<float> varHostData = {1, 1};
    std::vector<int32_t> outHostData(8, 0);
    float inputScaleValue = 1.0f;
    int32_t inputZeroPointValue = 1;
    float outputScaleValue = 2.0f;
    int32_t outputZeroPointValue = 1;
    double eps = 1e-5;

    // Create a self aclTensor.
    ret = CreateAclTensorNCHW(selfHostData, selfShape, &selfDeviceAddr, aclDataType::ACL_INT32, &self);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create a mean aclTensor.
    ret = CreateAclTensor(meanHostData, meanShape, &meanDeviceAddr, aclDataType::ACL_FLOAT, &mean);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create a var aclTensor.
    ret = CreateAclTensor(varHostData, varShape, &varDeviceAddr, aclDataType::ACL_FLOAT, &var);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create a weight aclTensor.
    ret = CreateAclTensor(weightHostData, weightShape, &weightDeviceAddr, aclDataType::ACL_FLOAT, &weight);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create a bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create an out aclTensor.
    ret = CreateAclTensorNCHW(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_INT32, &out);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create an inputScale aclScalar.
    aclScalar* inputScale = aclCreateScalar(&inputScaleValue, aclDataType::ACL_FLOAT);
    // Create an inputZeroPoint aclScalar.
    aclScalar* inputZeroPoint = aclCreateScalar(&inputZeroPointValue, aclDataType::ACL_INT32);
    // Create an outputScale aclScalar.
    aclScalar* outputScale = aclCreateScalar(&outputScaleValue, aclDataType::ACL_FLOAT);
    // Create an outputZeroPoint aclScalar.
    aclScalar* outputZeroPoint = aclCreateScalar(&outputZeroPointValue, aclDataType::ACL_INT32);

    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;

    // aclnnQuantizedBatchNorm API call example
    // 3. Call the CANN operator library API, which needs to be replaced with the actual API.
    // Call the first-phase API of aclnnQuantizedBatchNorm.
    ret = aclnnQuantizedBatchNormGetWorkspaceSize(
        self, mean, var, inputScale, inputZeroPoint, outputScale, outputZeroPoint, weight, bias, eps, out,
        &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantizedBatchNormGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Allocate device memory based on workspaceSize computed by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnQuantizedBatchNorm.
    ret = aclnnQuantizedBatchNorm(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantizedBatchNorm failed. ERROR: %d\n", ret); return ret);

    // 4. (Fixed writing) Wait until the task execution is complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host. Modify the configuration based on the API definition.
    auto size = GetShapeSize(outShape);
    printf("size is %ld", size);
    std::vector<int32_t> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
    }

    // 6. Release aclTensor and aclScalar. Modify the configuration based on the API definition.
    aclDestroyTensor(self);
    aclDestroyTensor(weight);
    aclDestroyTensor(bias);
    aclDestroyTensor(mean);
    aclDestroyTensor(var);
    aclDestroyTensor(out);
    aclDestroyScalar(inputScale);
    aclDestroyScalar(inputZeroPoint);
    aclDestroyScalar(outputScale);
    aclDestroyScalar(outputZeroPoint);

    // 7. Release device resources. Modify the configuration based on the API definition.
    aclrtFree(selfDeviceAddr);
    aclrtFree(meanDeviceAddr);
    aclrtFree(varDeviceAddr);
    aclrtFree(weightDeviceAddr);
    aclrtFree(biasDeviceAddr);
    aclrtFree(outDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}