aclnnInplaceQuantScatterV2

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	×
Atlas A2 training products/Atlas A2 inference products	×
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

A fusion of the Quantize operator and the Scatter operator. It first quantizes updates along the quantAxis: quantScales scales updates, and quantZeroPoints applies an offset. Then, the quantized values in updates are used to update the corresponding positions in selfRef one by one along the specified axis according to the index tensor indices. Compared with aclnnInplaceQuantScatter, this operator adds the roundMode input.

Function Prototype

Each operator uses a two-phase API. You must first call the "aclnnInplaceQuantScatterV2GetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call the "aclnnInplaceQuantScatterV2" API to perform the computation.

aclnnStatus aclnnInplaceQuantScatterV2GetWorkspaceSize(
  aclTensor       *selfRef,
  const aclTensor *indices,
  const aclTensor *updates,
  const aclTensor *quantScales,
  const aclTensor *quantZeroPoints,
  int64_t          axis,
  int64_t          quantAxis,
  int64_t          reduction,
  const char       *roundMode,
  uint64_t         *workspaceSize,
  aclOpExecutor   **executor);

aclnnStatus aclnnInplaceQuantScatterV2(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream);

aclnnInplaceQuantScatterV2GetWorkspaceSize

Parameters:

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-Contiguous Tensor
selfRef (aclTensor*)	Input\|Output	Source data tensor.	Supports empty tensors.	INT8, FLOAT8_E4M3FN, FLOAT_E5M2, HIFLOAT8	ND	3-8	√
indices (aclTensor*)	Input	Index tensor.	When the indices shape is 1D, the value range of indices is [0, selfRef.shape(axis) - updates.shape(axis)). When the indices shape is 2D, the first element of each indices entry ranges from [0, selfRef.shape(0)), and the second element of each indices entry ranges from [0, selfRef.shape(axis) - updates.shape(axis)).	INT32, INT64	ND	1, 2	√
updates (aclTensor*)	Input	Update data tensor.	The number of dimensions of updates must be the same as that of selfRef. The size of its first dimension equals the size of the first dimension of indices and is not greater than the size of the first dimension of selfRef. The size of its axis dimension is not greater than the size of the axis dimension of selfRef. The sizes of the remaining dimensions must be equal to the sizes of the corresponding dimensions of selfRef.	BFLOAT16, FLOAT16	ND	3-8	√
quantScales (aclTensor*)	Input	Quantization scaling tensor.	The number of elements in quantScales must equal the size of updates along the quantAxis axis.	BFLOAT16, FLOAT32	ND	1-8	√
quantZeroPoints (aclTensor*)	Input	Quantization offset tensor.	The number of elements in quantZeroPoints must equal the size of updates along the quantAxis axis.	BFLOAT16, INT32	ND	1–8	√
axis (int64_t)	Input	Specifies the axis on updates for performing the update.	Value range: [-len(updates.shape) + 1, -1) or [1, len(updates.shape) - 1).	-	-	-	-
quantAxis (int64_t)	Input	Axis on updates used for quantization.	Supports -1 or len(updates.shape) - 1.	-	-	-	-
reduction (int64_t)	Input	Specifies the data operation mode.	Currently, only the value 1 is supported.	-	-	-	-
roundMode (char*)	Input	Specifies the rounding mode in the Quantize formula for data conversion.	Supports {"rint", "round", "hybrid"} modes. When the data type of selfRef is INT8/FLOAT8_E4M3FN/FLOAT8_E5M2, only {"rint"} is supported. When the data type is HIFLOAT8, {"round", "hybrid"} are supported.	-	-	-	-
workspaceSize	Output	Returns the workspace size to be applied for on the device side.	-	-	-	-	-
executor	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	Returns 161001 if the passed parameter is a required input, output, or required attribute and is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The data types of selfRef, indices, updates, quantScales, and quantZeroPoints are not in the supported range.
		The data type combination of selfRef, indices, updates, quantScales, and quantZeroPoints is not in the supported range. For specific combinations, see Constraints.
		The number of dimensions of selfRef and updates is inconsistent.
		axis, quantAxis, or roundMode is not in the supported range.
		Unsupported product models.

aclnnInplaceQuantScatterV2

Parameters:

Parameter	Input/Output	Description
workspace	Input	Device-side workspace memory address.
workspaceSize	Input	Device-side workspace size, obtained from the first-phase API aclnnInplaceQuantScatterV2GetWorkspaceSize.
executor	Input	Operator executor that contains the operator computation flow.
stream	Input	Specifies the stream for task execution.

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnInplaceQuantScatterV2 defaults to a deterministic implementation.
The dimension of indices can only be 1D or 2D. If it is 2D, the size of the second dimension must be 2. Out-of-bounds indices are not supported and are not validated. The selfRef data segments mapped by indices must not overlap; otherwise, the results may vary across multiple executions due to multi-core concurrency.

The data type input combinations of selfRef, indices, updates, quantScales, and quantZeroPoints include:

Ascend 950PR/Ascend 950DT:

selfRef	indices	updates	quantScales	quantZeroPoints
INT8, FLOAT8_E4M3FN, FLOAT_E5M2, HIFLOAT8	INT32	BFLOAT16	BFLOAT16	BFLOAT16
INT8, FLOAT8_E4M3FN, FLOAT_E5M2, HIFLOAT8	INT64	BFLOAT16	BFLOAT16	BFLOAT16
INT8, FLOAT8_E4M3FN, FLOAT_E5M2, HIFLOAT8	INT32	FLOAT16	FLOAT32	INT32
INT8, FLOAT8_E4M3FN, FLOAT_E5M2, HIFLOAT8	INT64	FLOAT16	FLOAT32	INT32

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_quant_scatter_v2.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate for resource initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to apply for device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Copy host-side data to device memory using aclrtMemcpy.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Create an aclTensor by calling the aclCreateTensor API.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
  // Fill in the deviceId based on your actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the input and output. Customize the construction based on the API.
  std::vector<int64_t> selfRefShape = {1, 1, 32};
  std::vector<int64_t> indicesShape = {1};
  std::vector<int64_t> updatesShape = {1, 1, 32};
  std::vector<int64_t> quantScalesShape = {1, 1, 32};
  std::vector<int64_t> quantZeroPointsShape = {1, 1, 32};
  void* selfRefDeviceAddr = nullptr;
  void* indicesDeviceAddr = nullptr;
  void* updatesDeviceAddr = nullptr;
  void* quantScalesDeviceAddr = nullptr;
  void* quantZeroPointsDeviceAddr = nullptr;
  aclTensor* selfRef = nullptr;
  aclTensor* indices = nullptr;
  aclTensor* updates = nullptr;
  aclTensor* quantScales = nullptr;
  aclTensor* quantZeroPoints = nullptr;
  std::vector<int8_t> selfRefHostData{32, 0};
  std::vector<int32_t> indicesHostData{0};
  std::vector<float> updatesHostData{32, 1.0};
  std::vector<float> quantScalesHostData{32, 0.5};
  std::vector<float> quantZeroPointsHostData{32, 0.5};
  int64_t axis = -2;
  int64_t quantAxis = -1;
  int64_t reduction = 1;

  // Create the selfRef aclTensor.
  ret = CreateAclTensor(selfRefHostData, selfRefShape, &selfRefDeviceAddr, aclDataType::ACL_INT8, &selfRef);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the indices aclTensor.
  ret = CreateAclTensor(indicesHostData, indicesShape, &indicesDeviceAddr, aclDataType::ACL_INT32, &indices);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the updates aclTensor.
  ret = CreateAclTensor(updatesHostData, updatesShape, &updatesDeviceAddr, aclDataType::ACL_FLOAT16, &updates);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the quantScales aclTensor.
  ret = CreateAclTensor(quantScalesHostData, quantScalesShape, &quantScalesDeviceAddr, aclDataType::ACL_FLOAT,
                        &quantScales);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the quantZeroPoints aclTensor.
  ret = CreateAclTensor(quantZeroPointsHostData, quantZeroPointsShape, &quantZeroPointsDeviceAddr,
                        aclDataType::ACL_INT32, &quantZeroPoints);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // 3. Call the CANN operator library API. Replace with the specific API name.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  const char* roundMode = "rint";
  // Call the aclnnInplaceQuantScatterV2 first-phase API.
  ret = aclnnInplaceQuantScatterV2GetWorkspaceSize(selfRef, indices, updates, quantScales, quantZeroPoints, axis,
                                                 quantAxis, reduction, roundMode, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS,
            LOG_PRINT("aclnnInplaceQuantScatterV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Apply for device memory based on the workspaceSize calculated by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the aclnnInplaceQuantScatterV2 second-phase API.
  ret = aclnnInplaceQuantScatterV2(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnInplaceQuantScatterV2 failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronously wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device memory to the host. Modify based on the specific API definition.
  auto size = GetShapeSize(selfRefShape);
  std::vector<int8_t> resultData(size, 0);
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), selfRefDeviceAddr,
                    size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %d\n", i, resultData[i]);
  }

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(selfRef);
  aclDestroyTensor(indices);
  aclDestroyTensor(updates);
  aclDestroyTensor(quantScales);
  aclDestroyTensor(quantZeroPoints);

  // 7. Release device resources.
  aclrtFree(selfRefDeviceAddr);
  aclrtFree(indicesDeviceAddr);
  aclrtFree(updatesDeviceAddr);
  aclrtFree(quantScalesDeviceAddr);
  aclrtFree(quantZeroPointsDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();

  return 0;
}