aclnnQuantMatmulV2

Notice: This API will be deprecated in later versions. Use the latest aclnnQuantMatmulV5 API instead.

Applicable Products

Function

• Description: Performs quantized matrix multiplication with a maximum input dimension of 3. Similar APIs include aclnnMm (matrix multiplication supporting only 2D tensors as input) and aclnnBatchMatMul (3D matrix multiplication where the first dimension is the batch dimension).

• Formula:

Function Prototype

Each operator is divided into a two-phase API. The aclnnQuantMatmulV2GetWorkspaceSize API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the aclnnQuantMatmulV2 API is called to perform the computation.

aclnnStatus aclnnQuantMatmulV2GetWorkspaceSize(
  const aclTensor *x1,
  const aclTensor *x2,
  const aclTensor *bias,
  const aclTensor *deqScale,
  bool             adjX1,
  bool             adjX2,
  aclTensor       *out,
  uint64_t        *workspaceSize,
  aclOpExecutor   **executor)

aclnnStatus aclnnQuantMatmulV2(
  void              *workspace,
  uint64_t           workspaceSize,
  aclOpExecutor     *executor,
  const aclrtStream  stream)

aclnnQuantMatmulV2GetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Shape	Non-contiguous Tensor
  x1	Input	Input x1 in the formula.	When adjX1 is false, each dimension represents: (batch, m, k). When adjX1 is true, each dimension represents: (batch, k, m), where batch may be absent.	INT8	ND	2-3	-
  x2	Input	Input x2 in the formula.	When adjX2 is false, each dimension represents: (batch, k, n). When adjX2 is true, each dimension represents: (batch, n, k). batch may be absent, where k is consistent with k in the shape of x1.	INT8	ND	2-3	-
  bias	Input	Input bias in the formula.	A 1D tensor with shape (n,), where n is consistent with n of x2.	INT32	ND	1	-
  deqScale	Input	Represents the quantization parameter, which is the input deqScale in the formula.	The shape is 1-dimensional (t,), where t = align(n, 16), and n is consistent with the n of x2.	UINT64	ND	1	-
  adjX1	Input	Indicates whether the input shape of x1 includes transpose.	When adjX1 is false, each dimension represents: (batch, m, k). When adjX1 is true, each dimension represents: (batch, k, m), where batch may be absent.	bool	-	-	-
  adjX2	Input	Specifies whether the input shape of x2 includes transpose.	When adjX2 is false, each dimension represents: (batch, k, n). When adjX2 is true, each dimension represents: (batch, n, k), where batch may be absent.	bool	-	-	-
  out	Output	The output out in the formula.	(batch, m, n), where batch may be absent. Supports broadcasting of the batch dimensions of x1 and x2. The output batch is consistent with the batch after broadcasting. m and n are consistent with m of x1 and n of x2, respectively.	FLOAT16	ND	2-3	√
  workspaceSize	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
  executor	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The passed x1, x2, deqScale, or out is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	The data type or data format of x1, x2, bias, deqScale, or out is not within the supported range.
  		The shape of x1 or x2 is not 2D or 3D.
  		The shape of bias or deqScale is not 1D.
  		The size of deqScale is not equal to the n dimension of x2 rounded up to a multiple of 16.
  		bias exists and is an empty tensor where m and n are not 0 but k is 0.

aclnnQuantMatmulV2

• Parameters:

  Parameter	Input/Output	Description
  workspace	Input	Specifies the workspace memory address applied for on the Device side.
  workspaceSize	Input	Specifies the workspace size applied for on the Device side, which is obtained from the first-phase API aclnnQuantMatmulV2GetWorkspaceSize.
  executor	Input	Represents the op executor that contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic description:

  • Atlas A3 training products/Atlas A3 inference products, Atlas A2 training products/Atlas A2 inference products: aclnnQuantMatmulV2 defaults to a deterministic implementation.

Method for migrating this API to aclnnQuantMatmulV4:

• Inputs x1, x2, bias, adjX1, and adjX2 can be directly mapped to x1, x2, bias, transposeX1, and transposeX2 in the aclnnQuantMatmulV4 API.

• The input deqScale is an aclTensor of type UINT64, whose data type is consistent with scale in the aclnnQuantMatmulV4 API. The deqScale shape in the aclnnQuantMatmulV2 API is 1-dimensional (t,), where t = align(n, 16). The scale shape in the aclnnQuantMatmulV4 API is 1-dimensional (t,), where t = 1 or n. Directly call aclnnTransQuantParamV2 with the original FLOAT-type quantization parameters to output an aclTensor of data type UINT64 and shape (n,) (refer to the aclnnQuantMatmulV4 example), denoted as scale, which corresponds to scale in the aclnnQuantMatmulV4 API.

• The optional input offset/pertokenScaleOptional in the aclnnQuantMatmulV4 API is set to nullptr.

• The API parameters are set to aclnnQuantMatmulV4GetWorkspaceSize(x1, x2, scale, nullptr, nullptr, bias, adjX1, adjX2, out, workspaceSize, executor).

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <memory>
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul.h"
#include "aclnnop/aclnn_trans_quant_param.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      Finalize(deviceId, stream);\
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate: initialize resources.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to allocate device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Call aclrtMemcpy to copy data from the host side to the device-side memory.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Compute the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call the aclCreateTensor API to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream) {
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
}

int aclnnQuantMatmulV2Test(int32_t deviceId, aclrtStream &stream) {
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the inputs and outputs. Custom construction is required based on the API.
  std::vector<int64_t> x1Shape = {5, 2};
  std::vector<int64_t> x2Shape = {2, 3};
  std::vector<int64_t> biasShape = {3};
  std::vector<int64_t> outShape = {5, 3};
  void* x1DeviceAddr = nullptr;
  void* x2DeviceAddr = nullptr;
  void* biasDeviceAddr = nullptr;
  void* outDeviceAddr = nullptr;
  aclTensor* x1 = nullptr;
  aclTensor* x2 = nullptr;
  aclTensor* bias = nullptr;
  aclTensor* out = nullptr;
  std::vector<int8_t> x1HostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
  std::vector<int8_t> x2HostData = {1, 1, 1, 1, 1, 1};
  std::vector<int32_t> biasHostData = {1, 1, 1};
  std::vector<uint16_t> outHostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}; // Actually in float16 half-precision format.
  // Create the x1 aclTensor.
  ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
  std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
  std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the x2 aclTensor.
  ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
  std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
  std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the bias aclTensor.
  ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
  std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
  std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the out aclTensor.
  ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
  std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
  std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  bool adjX1 = false;
  bool adjX2 = false;

  // Convert the scale data from float type to the uint64_t type required by the hardware, and create the corresponding tensor as the input parameter deqScale.
  float scaleArray[3] = {1.0, 1.0, 1.0};
  uint64_t scaleSize = 3;
  float *offsetArray = nullptr;
  uint64_t offsetSize = 0;
  uint64_t *deqScaleArray = nullptr;
  uint64_t deqScaleSize = 0;
  ret = aclnnTransQuantParam(scaleArray, scaleSize, offsetArray, offsetSize, &deqScaleArray, &deqScaleSize);
  std::unique_ptr<uint64_t> deqScaleArrayPtr(deqScaleArray);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParam failed. ERROR: %d\n", ret); return ret);

  std::vector<uint64_t> deqScaleVector(deqScaleArray, deqScaleArray + deqScaleSize);
  std::vector<int64_t> deqScaleShape = {deqScaleSize};
  void* deqScaleDeviceAddr = nullptr;
  aclTensor* deqScale = nullptr;
  ret = CreateAclTensor(deqScaleVector, deqScaleShape, &deqScaleDeviceAddr, aclDataType::ACL_UINT64, &deqScale);
  std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> deqScaleTensorPtr(deqScale, aclDestroyTensor);
  std::unique_ptr<void, aclError (*)(void *)> deqScaleDeviceAddrPtr(deqScaleDeviceAddr, aclrtFree);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // 3. Call the CANN operator library API. Replace this with the specific API name.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  // Call the first-phase API of aclnnQuantMatmulV2.
  ret = aclnnQuantMatmulV2GetWorkspaceSize(x1, x2, bias, deqScale, adjX1, adjX2, out, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Apply for device memory based on the workspaceSize computed by the first-phase API.
  void* workspaceAddr = nullptr;
  std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    workspaceAddrPtr.reset(workspaceAddr);
  }
  // Call the second-phase API of aclnnQuantMatmulV2.
  ret = aclnnQuantMatmulV2(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV2 failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronously wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value, copy the result from the device memory to the host side, and modify based on the specific API definition.
  auto size = GetShapeSize(outShape);
  std::vector<uint16_t> resultData(size, 0); // fp16 data cannot be directly printed in C. Read it as uint16 and manually convert the binary to fp16.
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                    size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %ld\n", i, resultData[i]);
  }

  return ACL_SUCCESS;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
  // Set deviceId based on the actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = aclnnQuantMatmulV2Test(deviceId, stream);
  CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV2Test failed. ERROR: %d\n", ret); return ret);

  Finalize(deviceId, stream);
  return 0;
}