aclnnFlashAttentionScoreGrad

Applicable Products

Function

• Description: Computes the backward output of attention in training scenarios, i.e., the backward computation of aclnnFlashAttentionScore.

• Formula:

  The forward computation formula of attention is:

  $$Y=Dropout(Softmax(Mask(\frac{QK^T+pse}{\sqrt{d}}),atten\_mask),keep\_prob)V$$

  For ease of expression, variables $S$ and $P$ are used to represent the formulas:

  $$S=Mask(\frac{QK^T+pse}{\sqrt{d}},atten\_mask)$$ $$P=Dropout(Softmax(S),keep\_prob)$$ $$Y=PV$$

  The backward propagation formula for attention is:

  $$dV=P^TdY$$ $$dQ=\frac{((dS)*K)}{\sqrt{d}}$$ $$dK=\frac{((dS)^T*Q)}{\sqrt{d}}$$

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnFlashAttentionScoreGradGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call the "aclnnFlashAttentionScoreGrad" API to perform the computation.

aclnnStatus aclnnFlashAttentionScoreGradGetWorkspaceSize(
  const aclTensor   *query,
  const aclTensor   *keyIn,
  const aclTensor   *value,
  const aclTensor   *dy,
  const aclTensor   *pseShiftOptional,
  const aclTensor   *dropMaskOptional,
  const aclTensor   *paddingMaskOptional,
  const aclTensor   *attenMaskOptional,
  const aclTensor   *softmaxMaxOptional,
  const aclTensor   *softmaxSumOptional,
  const aclTensor   *softmaxInOptional,
  const aclTensor   *attentionInOptional,
  const aclIntArray *prefixOptional,
  double             scaleValue,
  double             keepProb,
  int64_t            preTokens,
  int64_t            nextTokens,
  int64_t            headNum,
  char              *inputLayout,
  int64_t            innerPrecise,
  int64_t            sparseMode,
  const aclTensor   *dqOut,
  const aclTensor   *dkOut,
  const aclTensor   *dvOut,
  const aclTensor   *dpseOut,
  uint64_t          *workspaceSize,
  aclOpExecutor    **executor)

aclnnStatus aclnnFlashAttentionScoreGrad(
  void             *workspace,
  uint64_t          workspaceSize,
  aclOpExecutor    *executor,
  const aclrtStream stream)

aclnnFlashAttentionScoreGradGetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  query	Input	Q in the formula.	The data type must be consistent with that of keyIn/value.	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  keyIn	Input	K in the formula.	The data type must be the same as that of query and value.	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  Value	Input	V in the formula.	The data type is the same as that of query/keyIn.	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  dy	Input	Input dY in the formula.	-	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  pseShiftOptional	Optional input.	pse in the formula.	The data type is the same as that of query.	FLOAT16, BFLOAT16, FLOAT32	ND	[B,N,Sq,Skv], [B,N,1,Skv], [1,N,Sq,Skv], [B,N,1024,Skv], [1,N,1024,Skv]	√
  dropMaskOptional	Optional input	Dropout in the formula.	-	UINT8	ND	0, 1	√
  paddingMaskOptional	Optional input	Reserved parameter, not currently used.	-	-	-	-	-
  attenMaskOptional	Optional input.	atten_mask in the formula.	A value of 1 indicates that the position does not participate in the computation, and 0 indicates that the position participates in the computation.	BOOL, UINT8	ND	[B,N,Sq,Skv], [B,1,Sq,Skv], [1,1,Sq,Skv], [Sq,Skv]	√
  softmaxMaxOptional	Optional input.	Intermediate output of the attention forward computation.	-	FLOAT	ND	[B,N,Sq,8]	√
  softmaxSumOptional	Optional input.	Intermediate output of the attention forward computation.	-	FLOAT	ND	[B,N,Sq,8]	√
  softmaxInOptional	Optional input	Intermediate output of the attention forward computation.	Reserved parameter, currently unused.	-	-	-	-
  attentionInOptional	Optional input.	Final output of the attention forward computation.	The data type and shape type are consistent with those of query.	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  prefixOptional	Optional input	Represents the N value of each batch in the prefix sparse computation scenario.	-	INT64	ND	0, 1	-
  scaleValue	Optional input.	In the formula, scale represents the scale factor.	-	DOUBLE	-	-	-
  keepProb	Optional input.	Represents the proportion of 1s in dropMaskOptional.	-	DOUBLE	-	-	-
  preTokens	Optional Input	Specifies the left boundary of the sliding window for sparse computation.	-	INT64	-	-	-
  nextTokens	optional input	Used for sparse computation. Indicates the right boundary of the sliding window.	-	INT64	-	-	-
  headNum	Input	Represents the number of heads per card, i.e., the N-axis length of the input query.	-	INT64	-	-	-
  inputLayout	Input.	Represents the data layout format of the input query, key, and value.	Supports BSH, SBH, BSND, and BNSD.	String	-	-	-
  innerPrecise	Optional input	Internal computation precision control.	Reserved.	INT64	-	-	-
  sparseMode	Optional input	Indicates the sparse mode.	Supports configuration values of 0, 1, 2, 3, 4, 5, and 6.	INT64	-	-	-
  dqOut	Output	dQ in the formula, representing the gradient of query.	-	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  dkOut	Output	dK in the formula, representing the gradient of keyIn.	-	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  dvOut	Output	dV in the formula represents the gradient of value.	-	FLOAT16, BFLOAT16, FLOAT32	ND	[BNSD], [BSND], [BSH], [SBH]	√
  dpseOut	Output	Gradient of d(pse).	Reserved parameter, currently unused.	-	-	-	-
  workspaceSize	Output	Returns the workspace size to be applied on the Device side.	-	-	-	-	-
  executor	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The input parameter is a mandatory input, output, or mandatory attribute, and is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	The data type of query, keyIn, value, dy, pseShiftOptional, dropMaskOptional, paddingMaskOptional, attenMaskOptional, softmaxMaxOptional, softmaxSumOptional, softmaxInOptional, attentionInOptional, dqOut, dkOut, or dvOut is not within the supported range.
  		The data format of query, keyIn, value, dy, pseShiftOptional, dropMaskOptional, paddingMaskOptional, attenMaskOptional, softmaxMaxOptional, softmaxSumOptional, softmaxInOptional, attentionInOptional, dqOut, dkOut, or dvOut is not within the supported range.

aclnnFlashAttentionScoreGrad

• Parameters

  Parameter	Input/Output	Description
  workspace	Input	Workspace memory address allocated on the device side.
  workspaceSize	Input	Workspace size allocated on the device side, obtained through the first-phase API aclnnFlashAttentionScoreGradGetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic computation:

  • aclnnFlashAttentionScoreGrad defaults to a non-deterministic implementation. It supports enabling deterministic computation through aclrtCtxSetSysParamOpt.

• When this API is used with PyTorch, ensure that the versions of CANN-related packages match those of PyTorch-related packages.

• The B (batchsize) of the input query, key, value, and dy must be equal.

• The D (head dimension) of the input query, key, and value must satisfy (qD == kD && kD >= vD).

• The inputLayout of the input query, key, value, and dy must be consistent.

• The data types of the input query, key, value, and pseShiftOptional must be consistent.

• The shapes of the input key and value must be consistent except for D. When the D sizes of query, key, and value are the same, the shapes of query and dy must be consistent.

• The N of the input query/dy and the N of key/value may differ, but they must be in a proportional relationship, i.e., Nq/Nkv must be a non-zero integer, with Nq in the value range of 1 to 256.

• Constraints on data shape, taking inputLayout BSND and BNSD as examples (BSH and SBH where H = N * D), where:

  • B: value range is 1 to 2M. When prefixOptional is present, B supports a maximum of 2K.

  • N: value range is 1 to 256.

  • S: value range is 1 to 1M.

  • D: value range is 1 to 768.

• The data layout formats of query, keyIn, and value support interpretation from multiple dimensions, where B (Batch) indicates the batch size of input samples, S (Seq-Length) indicates the sequence length of input samples, H (Head-Size) indicates the hidden dimension, N (Head-Num) indicates the number of heads, and D (Head-Dim) indicates the minimum unit size of the hidden layer, satisfying D = H / N.

• pseShiftOptional: If Sq is greater than 1024, and Sq and Skv of each batch have the same length, and the scenario is a lower triangular mask with sparseMode 0, 2, or 3, you can enable alibi positional encoding compression. In this case, only the last 1024 rows of the original PSE need to be input for memory optimization, i.e., alibi_compress = ori_pse[:, :, -1024:, :], as detailed below:

  • When each batch differs, the shape is BNHSkv (H = 1024).

  • When each batch is the same, the shape is 1NHSkv (H = 1024).

  • If this parameter is not used, nullptr can be passed in.

• innerPrecise: Currently, 0 and 1 are reserved configuration values, and 2 enables invalid row computation. This function avoids precision loss caused by entire-row masks during computation, but this configuration degrades performance. If the operator can determine that an invalid row scenario exists, it automatically enables invalid row computation, for example, when sparseMode is 3 and Sq > Skv.

• Constraints on sparseMode are as follows:

  • When the shape of all attenMaskOptional is less than 2048 and identical, it is recommended to use the default mode to reduce memory usage.

  • When configured as 1, 2, 3, or 5, the user-configured preTokens and nextTokens do not take effect.

  • When configured as 0 or 4, ensure that attenMaskOptional is consistent with the ranges of preTokens and nextTokens.

  • If you do not specify a value, 0 is recommended.

  • For details about different sparse modes, see Sparse Mode Description.

• In some scenarios, if the computation workload is too large, the operator execution may time out (aicore error, with errorStr: timeout or trap error). In this case, axis splitting is recommended. Note: The computation workload here is affected by parameters such as B, S, N, and D. Larger values result in a heavier computation workload.

• Constraints on the softmaxMax and softmaxSum parameters: The input format is fixed as [B, N, S, 8], except for the TND input format, which is [T, N, 8]. Note: T = B * S.

• The value of headNum must be consistent with the N value in the input Query.

In the band scenario, there must be an intersection between preTokens and nextTokens. When Sq in pseShiftOptional is greater than 1024 and the shape is BNHS or 1NHS, Sq and Skv must be of equal length. In the prefixOptional sparse computation scenario, i.e., sparseMode=5 or sparseMode=6, when Sq > Skv, the value range of N in prefix is [0, Skv]; when Sq <= Skv, the value range of N in prefix is [Skv-Sq, Skv].

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_flash_attention_score_grad.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

void PrintOutResult(std::vector<int64_t> &shape, void** deviceAddr) {
  auto size = GetShapeSize(shape);
  std::vector<float> resultData(size, 0);
  auto ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]),
                         *deviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("mean result[%ld] is: %f\n", i, resultData[i]);
  }
}

int Init(int32_t deviceId, aclrtContext* context, aclrtStream* stream) {
  // Boilerplate, resource initialization
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateContext(context, deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateContext failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetCurrentContext(*context);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetCurrentContext failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to apply for device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Call aclrtMemcpy to copy host-side data to device-side memory.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call the aclCreateTensor API to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
  // Fill in the deviceId based on your actual device.
  int32_t deviceId = 0;
  aclrtContext context;
  aclrtStream stream;
  auto ret = Init(deviceId, &context, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the input and output. Custom construction is required based on the API.
  std::vector<int64_t> qShape = {256, 1, 128};
  std::vector<int64_t> kShape = {256, 1, 128};
  std::vector<int64_t> vShape = {256, 1, 128};
  std::vector<int64_t> dxShape = {256, 1, 128};
  std::vector<int64_t> attenmaskShape = {256, 256};
  std::vector<int64_t> softmaxMaxShape = {1, 1, 256, 8};
  std::vector<int64_t> softmaxSumShape = {1, 1, 256, 8};
  std::vector<int64_t> attentionInShape = {256, 1, 128};

  std::vector<int64_t> dqShape = {256, 1, 128};
  std::vector<int64_t> dkShape = {256, 1, 128};
  std::vector<int64_t> dvShape = {256, 1, 128};

  void* qDeviceAddr = nullptr;
  void* kDeviceAddr = nullptr;
  void* vDeviceAddr = nullptr;
  void* dxDeviceAddr = nullptr;
  void* attenmaskDeviceAddr = nullptr;
  void* softmaxMaxDeviceAddr = nullptr;
  void* softmaxSumDeviceAddr = nullptr;
  void* attentionInDeviceAddr = nullptr;
  void* dqDeviceAddr = nullptr;
  void* dkDeviceAddr = nullptr;
  void* dvDeviceAddr = nullptr;

  aclTensor* q = nullptr;
  aclTensor* k = nullptr;
  aclTensor* v = nullptr;
  aclTensor* dx = nullptr;
  aclTensor* pse = nullptr;
  aclTensor* dropMask = nullptr;
  aclTensor* padding = nullptr;
  aclTensor* attenmask = nullptr;
  aclTensor* softmaxMax = nullptr;
  aclTensor* softmaxSum = nullptr;
  aclTensor* softmaxIn = nullptr;
  aclTensor* attentionIn = nullptr;
  aclTensor* dq = nullptr;
  aclTensor* dk = nullptr;
  aclTensor* dv = nullptr;
  aclTensor* dpse = nullptr;

  std::vector<float> qHostData(32768, 1);
  std::vector<float> kHostData(32768, 1);
  std::vector<float> vHostData(32768, 1);
  std::vector<float> dxHostData(32768, 1);
  std::vector<uint8_t> attenmaskHostData(65536, 0);
  std::vector<float> softmaxMaxHostData(2048, 3.0);
  std::vector<float> softmaxSumHostData(2048, 3.0);
  std::vector<float> attentionInHostData(32768, 1);
  std::vector<float> dqHostData(32768, 0);
  std::vector<float> dkHostData(32768, 0);
  std::vector<float> dvHostData(32768, 0);

  ret = CreateAclTensor(qHostData, qShape, &qDeviceAddr, aclDataType::ACL_FLOAT, &q);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(kHostData, kShape, &kDeviceAddr, aclDataType::ACL_FLOAT, &k);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(vHostData, vShape, &vDeviceAddr, aclDataType::ACL_FLOAT, &v);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(dxHostData, dxShape, &dxDeviceAddr, aclDataType::ACL_FLOAT, &dx);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(attenmaskHostData, attenmaskShape, &attenmaskDeviceAddr, aclDataType::ACL_UINT8, &attenmask);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(softmaxMaxHostData, softmaxMaxShape, &softmaxMaxDeviceAddr, aclDataType::ACL_FLOAT, &softmaxMax);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(softmaxSumHostData, softmaxSumShape, &softmaxSumDeviceAddr, aclDataType::ACL_FLOAT, &softmaxSum);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(attentionInHostData, attentionInShape, &attentionInDeviceAddr, aclDataType::ACL_FLOAT, &attentionIn);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(dqHostData, dqShape, &dqDeviceAddr, aclDataType::ACL_FLOAT, &dq);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(dkHostData, dkShape, &dkDeviceAddr, aclDataType::ACL_FLOAT, &dk);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(dvHostData, dvShape, &dvDeviceAddr, aclDataType::ACL_FLOAT, &dv);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  std::vector<int64_t> prefixOp = {0};
  aclIntArray *prefix = aclCreateIntArray(prefixOp.data(), 1);
  double scaleValue = 0.088388;
  double keepProb = 1;
  int64_t preTokens = 65536;
  int64_t nextTokens = 65536;
  int64_t headNum = 1;
  int64_t innerPrecise = 0;
  int64_t sparseMode = 0;

  char layOut[5] = {'S', 'B', 'H', 0};

  // 3. Call the CANN operator library API. Modify it to the specific API name.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;

  // Call the first-phase API of aclnnFlashAttentionScoreGrad.
  ret = aclnnFlashAttentionScoreGradGetWorkspaceSize(q, k, v, dx, pse, dropMask, padding,
            attenmask, softmaxMax, softmaxSum, softmaxIn, attentionIn, prefix,
            scaleValue, keepProb, preTokens, nextTokens, headNum, layOut, innerPrecise, sparseMode,
            dq, dk, dv, dpse, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFlashAttentionScoreGradGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);

  // Apply for device memory based on the workspaceSize calculated by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }

  // Call the second-phase API of aclnnFlashAttentionScoreGrad.
  ret = aclnnFlashAttentionScoreGrad(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFlashAttentionScoreGrad failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronize and wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify based on the specific API definition.
  PrintOutResult(dqShape, &dqDeviceAddr);
  PrintOutResult(dkShape, &dkDeviceAddr);
  PrintOutResult(dvShape, &dvDeviceAddr);

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(q);
  aclDestroyTensor(k);
  aclDestroyTensor(v);
  aclDestroyTensor(dx);
  aclDestroyTensor(attenmask);
  aclDestroyTensor(softmaxMax);
  aclDestroyTensor(softmaxSum);
  aclDestroyTensor(attentionIn);
  aclDestroyTensor(dq);
  aclDestroyTensor(dk);
  aclDestroyTensor(dv);

  // 7. Release device resources.
  aclrtFree(qDeviceAddr);
  aclrtFree(kDeviceAddr);
  aclrtFree(vDeviceAddr);
  aclrtFree(dxDeviceAddr);
  aclrtFree(attenmaskDeviceAddr);
  aclrtFree(softmaxMaxDeviceAddr);
  aclrtFree(softmaxSumDeviceAddr);
  aclrtFree(attentionInDeviceAddr);
  aclrtFree(dqDeviceAddr);
  aclrtFree(dkDeviceAddr);
  aclrtFree(dvDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtDestroyContext(context);
  aclrtResetDevice(deviceId);
  aclFinalize();

  return 0;
}