aclnnFusedMatmul

Applicable Products

Function

• Description: Fuses matrix multiplication with general vector computation.

• Formula:

  $$y = OP((x1 @ x2 + bias), x3)$$

  The OP type is defined by the fusedOpType input and supports the following:

  add operation:

  $$y=(x1 @ x2 + bias) + x3$$

  mul operation:

  $$y=(x1 @ x2 + bias) ∗ x3$$

  gelu_tanh operation:

  $$y = gelu\_tanh(x1 @ x2 + bias)$$

  gelu_erf operation:

  $$y = gelu\_erf(x1 @ x2 + bias)$$

  relu operation:

  $$y = relu(x1 @ x2 + bias)$$

Function Prototype

Each operator is divided into a two-phase API. The "aclnnFusedMatmulGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnFusedMatmul" API is called to perform the computation.

aclnnStatus aclnnFusedMatmulGetWorkspaceSize(
  const aclTensor *x1,
  const aclTensor *x2,
  const aclTensor *bias,
  const aclTensor *x3,
  const char      *fusedOpType,
  int8_t           cubeMathType,
  const aclTensor *y,
  uint64_t        *workspaceSize,
  aclOpExecutor   **executor)

aclnnStatus aclnnFusedMatmul(
  void            *workspace,
  uint64_t         workspaceSize,
  aclOpExecutor   *executor,
  aclrtStream      stream)

aclnnFusedMatmulGetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension	Non-Contiguous
  x1	Input	Represents the first matrix of the matrix multiplication, corresponding to x1 in the formula.	The data type must satisfy the data type deduction rules with x2 (see Deduction Relationship).	FLOAT16, BFLOAT16, FLOAT32	ND	2	√
  x2	Input	Represents the second matrix of the matrix multiplication, corresponding to x2 in the formula.	The data type must satisfy the data type deduction rules with x1 (see Deduction Relationship).	The data type must be consistent with x1.	ND	2	√
  bias	Input	Represents the bias term, corresponding to bias in the formula.	Takes effect only when fusedOpType is "", "relu", "add", or "mul". Pass a null pointer in other cases.	FLOAT16, BFLOAT16, FLOAT32	ND	1-2	√
  x3	Input	Represents the second matrix of the fused operation, corresponding to x3 in the formula.	-	The data type must be consistent with x1.	ND	2	√
  y	Output	Represents the output matrix of the computation, corresponding to y in the formula.	The data type must be consistent with the data type derived from x1 and x2 (see Deduction Relationship).	FLOAT16, BFLOAT16, FLOAT32	ND	2	√
  cubeMathType	Input	Specifies the computation logic of the Cube unit.	If the input data types have a deduction relationship, this parameter defaults to processing the data type after deduction. Supported enumeration values are as follows: 0: KEEP_DTYPE, keeps the input data type for computation. 1: ALLOW_FP32_DOWN_PRECISION, allows down-precision computation of the input data. 2: USE_FP16, allows down-precision of the input to FLOAT16 for computation. 3: USE_HF32, allows down-precision of the input to HFLOAT32 for computation. 4: USE_FP32_ADD, uses high-precision mode for computation.	INT8	-	-	-
  fusedOpType	Input	Specifies the fusion mode supported by the Matmul operator, which corresponds to OP in the formula.	The fusion mode value must be one of "" (indicating no fusion), "add", "mul", "gelu_erf", "gelu_tanh", or "relu".	STRING	-	-	-
  workspaceSize	Output	Returns the workspace size that the user needs to apply for on the Device side.	-	-	-	-	-
  executor	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

  • Ascend 950PR/Ascend 950DT:

    • cubeMathType=1: When the input data type is FLOAT32, it is converted to HFLOAT32 for computation. When the input is of other data types, no processing is performed.

    • cubeMathType=2: This option is not supported when the input data type is BFLOAT16.

    • cubeMathType=3: When the input data type is FLOAT32, it will be converted to HFLOAT32 for computation. This option is not supported when the input is of other data types.

    • No processing is performed when cubeMathType is set to 4.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Return Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	x1, x2, and y are null pointers.
  		When fusedOpType is add or mul, x3 is a null pointer.
  		When fusedOpType is gelu_tanh or gelu_erf, bias is not a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	The data type of x1 and x2 is not within the supported range.
  		The data format of x1, x2, or y is not within the supported range.
  		The dimensions of x1 and x2 are not two-dimensional.
  		When fusedOpType is add or mul, the shape of x3 does not match the output shape.
  		The passed fusedOpType is not one of "", "add", "mul", "gelu_tanh", "gelu_erf", or "relu".
  		Data type deduction cannot be performed on x1 and x2.
  		When the passed fusedOpType is one of "", "add", "mul", or "relu", and the input data type is float32, cubeMathType supports only 3.

aclnnFusedMatmul

• Parameters:

  Parameter	Input/Output	Description
  workspace	Input	Workspace memory address applied for on the device side.
  workspaceSize	Input	Workspace size applied for on the device side, obtained from the first-phase API aclnnFusedMatmulGetWorkspaceSize.
  Executor	Input	Op executor, which contains the operator computation flow.
  Stream	Input	Specifies the stream for task execution.

• Return value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic description:

• Atlas training products, Atlas inference products: aclnnFusedMatmul defaults to a deterministic implementation.

• When fusedOpType is set to "gelu_erf" or "gelu_tanh", the data types of x1, x2, and x3 must be BFLOAT16 or FLOAT16. When fusedOpType is "", "relu", "add", or "mul", the data types of x1, x2, and x3 must be FLOAT32 (cubeMathType only supports 3), BFLOAT16, or FLOAT16.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_fused_matmul.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate for resource initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to allocate device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Call aclrtMemcpy to copy data from the host to the device memory.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Compute the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call aclCreateTensor to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
  // Set deviceId based on the actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the input and output. Custom construction is required based on the API interface.
  std::vector<int64_t> xShape = {16, 32};
  std::vector<int64_t> x2Shape = {32, 16};
  std::vector<int64_t> x3Shape = {16, 16};
  std::vector<int64_t> yShape = {16, 16};
  void* xDeviceAddr = nullptr;
  void* x2DeviceAddr = nullptr;
  void* x3DeviceAddr = nullptr;
  void* yDeviceAddr = nullptr;
  aclTensor* x = nullptr;
  aclTensor* x2 = nullptr;
  aclTensor* x3 = nullptr;
  aclTensor* y = nullptr;
  std::vector<float> xHostData(512, 1);
  std::vector<float> x2HostData(512, 1);
  std::vector<float> x3HostData(256, 1);
  std::vector<float> yHostData(256, 0);
  // Create the x aclTensor.
  ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the x2 aclTensor.
  ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_FLOAT16, &x2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the x3 aclTensor.
  ret = CreateAclTensor(x3HostData, x3Shape, &x3DeviceAddr, aclDataType::ACL_FLOAT16, &x3);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // Create the y aclTensor.
  ret = CreateAclTensor(yHostData, yShape, &yDeviceAddr, aclDataType::ACL_FLOAT16, &y);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // 3. Call the CANN operator library API. Replace it with the specific API name.
  int8_t cubeMathType = 0;
  const char* fusedOpType = "add";
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor = nullptr;
  // Call the first-phase API of aclnnFusedMatmul.
  ret = aclnnFusedMatmulGetWorkspaceSize(x, x2, nullptr, x3, fusedOpType, cubeMathType, y, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFusedMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Allocate device memory based on the workspaceSize computed by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the second-phase API of aclnnFusedMatmul.
  ret = aclnnFusedMatmul(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFusedMatmul failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronize and wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device side memory to the host side. Modify based on the specific API definition.
  auto size = GetShapeSize(yShape);
  std::vector<float> resultData(size, 0);
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), yDeviceAddr,
                    size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
  }

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(x);
  aclDestroyTensor(x2);
  aclDestroyTensor(x3);
  aclDestroyTensor(y);

  // 7. Release device resources. Modify based on the specific API definition.
  aclrtFree(xDeviceAddr);
  aclrtFree(x2DeviceAddr);
  aclrtFree(x3DeviceAddr);
  aclrtFree(yDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}