aclnnMultiScaleDeformableAttnFunction

Applicable Products

Function

• Description: Traverses different sampling points on feature maps of different sizes using parameters such as sampling locations, attention weights, mapped value features, the starting index positions of multi-scale features, and the spatial sizes of multi-scale feature maps (to convert sampling locations from normalized values to absolute positions).

• Formula:

  Maps the normalized coordinates $(u,v)\in[0,1]$ of a sampling point to the pixel coordinate system of the $\ell$-th feature map:

  $$x = u \cdot W_\ell - 0.5, \qquad y = v \cdot H_\ell - 0.5$$

  Determine which four integer grid points the sampling point falls between:

  $$x_0 = \lfloor x \rfloor,\quad x_1 = x_0 + 1,\qquad y_0 = \lfloor y \rfloor,\quad y_1 = y_0 + 1$$

  Calculate the offset of the sampling point relative to the top-left grid point for interpolation weights:

  $$\alpha_x = x - x_0, \qquad \alpha_y = y - y_0$$

  Calculate the bilinear interpolation weights. The sum of the four neighboring points is 1.

  $$\begin{aligned} w_{00} &= (1-\alpha_y)(1-\alpha_x), \\ w_{10} &= (1-\alpha_y)\alpha_x, \\ w_{01} &= \alpha_y(1-\alpha_x), \\ w_{11} &= \alpha_y\alpha_x \end{aligned}$$

  The feature vector (length $D$) corresponding to the sampling point is computed as:

  $$\operatorname{bilinear}(V;\,b,h,\ell,x,y) = w_{00} \, V_{b,\ell,y_0,x_0,h,:} + w_{10} \, V_{b,\ell,y_0,x_1,h,:} + w_{01} \, V_{b,\ell,y_1,x_0,h,:} + w_{11} \, V_{b,\ell,y_1,x_1,h,:}$$

  The bilinear sampling results from all levels and all sampling points are weighted and summed to obtain the final output:

  $$O_{b,q,h,:} = \sum_{\ell=0}^{L-1} \sum_{p=0}^{N_p-1} A_{b,q,h,\ell,p} \cdot \operatorname{bilinear}\!\left(V;\,b,h,\ell, x_{b,q,h,\ell,p}, y_{b,q,h,\ell,p}\right)$$

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call the "aclnnMultiScaleDeformableAttnFunction" API to perform the computation.

aclnnStatus aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize(
    const aclTensor* value,
    const aclTensor* spatialShape,
    const aclTensor* levelStartIndex,
    const aclTensor* location,
    const aclTensor* attnWeight,
    aclTensor*       output,
    uint64_t*        workspaceSize,
    aclOpExecutor**  executor)

aclnnStatus aclnnMultiScaleDeformableAttnFunction(
    void*          workspace,
    uint64_t       workspaceSize,
    aclOpExecutor* executor,
    aclrtStream    stream)

aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize

• Parameters:

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  Value	Input	Feature values of the feature map.	-	FLOAT, FLOAT16, BFLOAT16	ND	(bs, num_keys, num_heads, channels) where bs is the batch size, num_keys is the size of the feature map, num_heads is the number of heads, and channels is the dimension of the feature map.	√
  spatialShape	Input	Stores the height and width of each scale's feature map.	-	INT32, INT64	ND	(num_levels, 2) where num_levels is the number of feature maps, and 2 represents H and W respectively.	√
  levelStartIndex	Input	The starting index of each feature map.	-	INT32, INT64	ND	(num_levels)	√
  location	Input	Sampling point position tensor, which stores the coordinate position of each sampling point.	The data type must be consistent with value.	FLOAT, FLOAT16, BFLOAT16	ND	(bs, num_queries, num_heads, num_levels, num_points, 2) Where num_queries is the number of queries, num_points is the number of sampling points, and 2 represents y and x respectively.	√
  attnWeight	Input	Sampling point weight tensor.	The data type is the same as that of value.	FLOAT, FLOAT16, BFLOAT16	ND	(bs, num_queries, num_heads, num_levels, num_points)	√
  output	Output	The output computed by the operator.	The data type is the same as that of value.	FLOAT, FLOAT16, BFLOAT16	ND	(bs, num_queries, num_heads * channels)	√
  workspaceSize	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
  Executor	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

  • Atlas inference series products: BFLOAT16 data type is not supported.

• Return value:

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The input or output passed in is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	The data type of the input or output is not within the supported range.
  		The data types of the input and output are inconsistent.
  		The shape of value is not 4-dimensional.
  		spatialShape is not 2-dimensional.
  		levelStartIndex is not 1-dimensional.
  		location is not 6-dimensional.
  		attnWeight is not 5-dimensional.
  		The last axis of spatialShape is not 2.
  		The last axis of location is not 2.
  		The constraints section is not met.

aclnnMultiScaleDeformableAttnFunction

• Parameters:

  Parameter	Input/Output	Description
  workspace	Input	Address of the workspace memory applied on the Device side.
  workspaceSize	Input	Size of the workspace applied on the Device side, obtained by the first-phase API aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation process.
  stream	Input	Specifies the stream that executes the task.

• Return value:

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic computation:

  • aclnnMultiScaleDeformableAttnFunction defaults to a deterministic implementation.

• Atlas inference products:

  • channels % 32 = 0, and channels <= 256.

  • Query count: 32 <= num_queries < 500000.

  • Feature map count: num_levels <= 16.

  • Head count: num_heads = [2, 4, 8].

  • Sampling point count: num_points = [4, 8].

• Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

  • Channel count: channels%8 = 0, and channels <= 256.

  • The number of queries: 32 <= num_queries < 500000.

  • The number of feature maps: num_levels <= 16.

  • The number of heads: num_heads <= 16.

  • The number of sampling points: num_points <= 16.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_multi_scale_deformable_attn_function.h"

#define CHECK_RET(cond, return_expr) \
    do {                               \
        if (!(cond)) {                   \
        return_expr;                   \
        }                                \
    } while (0)

#define LOG_PRINT(message, ...)     \
    do {                              \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
  }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                    aclDataType dataType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for memory on the device side.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Call aclrtMemcpy to copy data from the host side to the device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
       strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

int main() {
    // 1. (Boilerplate) Initialize the device/stream. Refer to the acl external API list.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check based on your needs.
    CHECK_RET(ret == 0, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
    // 2. Construct the input and output. Customize the construction based on the API.
    std::vector<int64_t> valueShape = {1, 1, 2, 32};
    std::vector<int64_t> spatialShapeShape = {1, 2};
    std::vector<int64_t> levelStartIndexShape = {1};
    std::vector<int64_t> locationShape = {1, 32, 2, 1, 4, 2};
    std::vector<int64_t> attnWeightShape = {1, 32, 2, 1, 4};
    std::vector<int64_t> outputShape = {1, 32, 64};
    void* valueDeviceAddr = nullptr;
    void* spatialShapeDeviceAddr = nullptr;
    void* levelStartIndexDeviceAddr = nullptr;
    void* locationDeviceAddr = nullptr;
    void* attnWeightDeviceAddr = nullptr;
    void* outputDeviceAddr = nullptr;
    aclTensor* value = nullptr;
    aclTensor* spatialShape = nullptr;
    aclTensor* levelStartIndex = nullptr;
    aclTensor* location = nullptr;
    aclTensor* attnWeight = nullptr;
    aclTensor* output = nullptr;
    std::vector<float> valueHostData = {static_cast<float>(GetShapeSize(locationShape)), 1};
    std::vector<float> spatialShapeHostData = {1, 1};
    std::vector<float> levelStartIndexHostData = {0};
    std::vector<float> locationHostData(static_cast<float>(GetShapeSize(locationShape)), 0);
    std::vector<float> attnWeightHostData = {static_cast<float>(GetShapeSize(attnWeightShape)), 1};
    std::vector<float> outputHostData = {static_cast<float>(GetShapeSize(outputShape)), 1};
    // value aclTensor
    ret = CreateAclTensor(valueHostData, valueShape, &valueDeviceAddr, aclDataType::ACL_FLOAT, &value);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the spatialShape aclTensor.
    ret = CreateAclTensor(spatialShapeHostData, spatialShapeShape, &spatialShapeDeviceAddr, aclDataType::ACL_INT32, &spatialShape);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the levelStartIndex aclTensor.
    ret = CreateAclTensor(levelStartIndexHostData, levelStartIndexShape, &levelStartIndexDeviceAddr, aclDataType::ACL_INT32, &levelStartIndex);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the location aclTensor.
    ret = CreateAclTensor(locationHostData, locationShape, &locationDeviceAddr, aclDataType::ACL_FLOAT, &location);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the attnWeight aclTensor.
    ret = CreateAclTensor(attnWeightHostData, attnWeightShape, &attnWeightDeviceAddr, aclDataType::ACL_FLOAT, &attnWeight);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the output aclTensor.
    ret = CreateAclTensor(outputHostData, outputShape, &outputDeviceAddr, aclDataType::ACL_FLOAT, &output);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // 3. Call the CANN operator library API. Replace it with the specific API.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnMultiScaleDeformableAttnFunction.
    ret = aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize(value, spatialShape, levelStartIndex, location, attnWeight, output,
                                                                &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnMultiScaleDeformableAttnFunctionGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Allocate device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
    }
    // Call the second-phase API of aclnnMultiScaleDeformableAttnFunction.
    ret = aclnnMultiScaleDeformableAttnFunction(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnMultiScaleDeformableAttnFunction failed. ERROR: %d\n", ret); return ret);
    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
    // 5. Obtain the output value and copy the result from the device memory to the host. Modify based on the specific API definition.
    auto size = GetShapeSize(outputShape);
    std::vector<float> resultData(size, 0);
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outputDeviceAddr, size * sizeof(float),
                      ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
    }

    // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
    aclDestroyTensor(value);
    aclDestroyTensor(spatialShape);
    aclDestroyTensor(levelStartIndex);
    aclDestroyTensor(location);
    aclDestroyTensor(attnWeight);
    aclDestroyTensor(output);

    // Release device resources. Modify based on the specific API definition.
    aclrtFree(valueDeviceAddr);
    aclrtFree(spatialShapeDeviceAddr);
    aclrtFree(levelStartIndexDeviceAddr);
    aclrtFree(locationDeviceAddr);
    aclrtFree(attnWeightDeviceAddr);
    aclrtFree(outputDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}