aclnnClippedSwiglu

Product Support

Product	Supported
Atlas A3 training series products/Atlas A3 inference series products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference series products	×
Atlas training series products	×

Function

Description: Serves as a clipped Swish-gated linear unit (SwiGLU) activation function that computes the clipped SwiGLU of input tensor x. Compared with aclnnSwiGlu, this API introduces additional parameters — groupIndex, alpha, limit, bias, and interleaved — to support the SwiGLU variant used in GPT-OSS models and grouped scenarios in MoE models.
Formula:

For a given input tensor x with shape [a,b,c,d,e,f,g...], aclnnClippedSwiglu performs the following operations:
1. Merge the axes of x according to the input parameter dim. The merged shape is [pre,cut,after], where cut is the target dimension to be split into two tensors. The split modes include front-back split and even-odd split. pre and after can be 1. For example, if dim is 3, the shape of x after axis merging is [a * b * c, d, e * f * g * ...]. Since elements along the after axis are contiguous and operations are element-wise, the cut and after axes are further merged, resulting in a final shape of [pre,cut].
2. Filter the pre axis of x based on the input parameter group_index using the following formula:
  $sum = \text{Sum}(group\_index)$ $x = x[ : sum, : ]$
  Where sum is the sum of all elements in group_index. This step is skipped when group_index is not provided.
3. Split x based on the input parameter interleaved using the following formula:
  
  When interleaved is set to true, it indicates odd-even split:
  $A = x[ : , : : 2]$ $B = x[ : , 1 : : 2]$
  When interleaved is set to false, it indicates front-back split:
  $h = x.shape[1] // 2$ $A = x[ : , : h]$ $B = x[ : , h : ]$
4. Compute the clipped SwiGLU variant based on the input parameters alpha, limit, and bias using the following formula:
  $A = A.clamp(min=None, max=limit)$ $B = B.clamp(min=-limit, max=limit)$ $y\_glu = A * sigmoid(alpha * A)$ $y = y\_glu * (B + bias)$
5. Reshape the output tensor y to match the original number of dimensions of x. The size along the dim axis is half that of x, while all other dimensions remain the same as x.

Prototype

Each operator has two-phase API calls. First, aclnnClippedSwigluGetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, aclnnClippedSwiglu is called to perform computation.

aclnnStatus aclnnClippedSwigluGetWorkspaceSize(
    const aclTensor *x, 
    const aclTensor *groupIndexOptional, 
    int64_t          dim, 
    double           alpha, 
    double           limit, 
    double           bias, 
    bool             interleaved, 
    const aclTensor *out, 
    uint64_t        *workspaceSize, 
    aclOpExecutor   **executor)

aclnnStatus aclnnClippedSwiglu(
    void          *workspace, 
    uint64_t       workspaceSize, 
    aclOpExecutor *executor, 
    aclrtStream    stream)

aclnnClippedSwigluGetWorkspaceSize

Parameters

Name	Input/Output	Description	Usage Notes	Data Type	Data Format	Shape	Non-contiguous Tensor
x	Input	x in the formula.	Null pointers are not supported. The tensor rank must be greater than 0, and the size of the dim dimension must be even.	FLOAT, FLOAT16, BFLOAT16	ND	1-8	-
groupIndexOptional	Input	group_index in the formula, indicating how the input is grouped.	Null pointers are supported. If the pointer is not null, the tensor must be a 1D tensor with non-negative elements. The i-th element indicates the batch size of x to be processed for the i-th group.	INT64	ND	1	-
dim	Input	dim in the formula, indicating the dimension index used to merge and split tensor x.	The value range is [–x.dim(), x.dim() – 1].	INT64	-	-	-
alpha	Input	alpha in the formula, indicating the scaling factor used in the variant SwiGLU computation.	The recommended value is 1.702.	DOUBLE	-	-	-
limit	Input	limit in the formula, indicating the clipping threshold used in the variant SwiGLU computation.	The recommended value is 7.0.	DOUBLE	-	-	-
bias	Input	bias in the formula, indicating the bias parameter used in the variant SwiGLU computation.	The recommended value is 1.0.	DOUBLE	-	-	-
interleaved	Input	interleaved in the formula, specifying the split mode for x.	true: even-odd split; false: front-back split.	BOOL	-	-	-
out	Output	y in the formula.	Null pointers are not supported. The output shape is the same as x, except that the dim axis is halved.	FLOAT, FLOAT16, BFLOAT16	ND	1-8	-
workspaceSize	Output	Size of the workspace to be allocated on the device.	-	-	-	-	-
executor	Output	Operator executor, containing the operator computation flow.	-	-	-	-	-

Returns

aclnnStatus: status code. For details, see aclnn Return Codes.

The first-phase API implements input parameter verification. The following errors may be thrown.

Return	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The passed x or y is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The input or output data type is not supported.
		The input or output tensor rank is not supported.
		dim is out of the valid range.

aclnnClippedSwiglu

Parameters

Name	Input/Output	Description
workspace	Input	Address of the workspace to be allocated on the device.
workspaceSize	Input	Size of the workspace to be allocated on the device, obtained by calling the first-phase API aclnnClippedSwigluGetWorkspaceSize.
executor	Input	Operator executor, containing the operator computation flow.
stream	Input	Stream for executing the task.

Returns

aclnnStatus: status code. For details, see aclnn Return Codes.

Constraints

Deterministic computation: aclnnClippedSwiglu defaults to a deterministic implementation. The non-deterministic implementation is not supported, and configuration changes via deterministic computation settings will not take effect.

Example

The following example is for reference only. For details, see Compilation and Running Sample.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_clipped_swiglu.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // (Boilerplate) Initialize ACL.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate memory on the device.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy the data on the host to the memory on the device.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of the contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

int main()
{
    // 1. (Boilerplate) Initialize the device and stream. For details, see the ACL API manual.
    // Set the device ID (deviceId) based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs based on API definitions.
    std::vector<int64_t> xShape = {2, 32};
    std::vector<int64_t> groupIndexShape = {1};
    std::vector<int64_t> outShape = {2, 16};
    void* xDeviceAddr = nullptr;
    void* groupIndexDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclTensor* groupIndex = nullptr;
    aclTensor* out = nullptr;
    std::vector<float> xHostData = {0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
                             22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
                             44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63};
    std::vector<int64_t> groupIndexData = {1};
    std::vector<float> outHostData = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                               0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

    int dim = -1;
    float alpha = 1.0;
    float limit = 7.0;
    float bias = 1.702;
    bool interleaved = true;
    // Create an x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create a groupIndex aclTensor.
    ret = CreateAclTensor(groupIndexData, groupIndexShape, &groupIndexDeviceAddr, aclDataType::ACL_INT64, &groupIndex);
    // Create an out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT, &out);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // 3. Call the CANN operator library API. Modify the API as required.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnClippedSwiglu.
    ret = aclnnClippedSwigluGetWorkspaceSize(
        x, groupIndex, dim, alpha, limit, bias, interleaved, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnClippedSwigluGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Allocate device memory based on workspaceSize computed by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnClippedSwiglu.
    ret = aclnnClippedSwiglu(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnClippedSwiglu failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize the stream and wait for task completion.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device to the host. Modify the code based on the API definition.
    auto size = GetShapeSize(outShape);
    std::vector<float> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
    }
    // 6. Destroy aclTensor and aclScalar. Modify the code based on the API definition.
    aclDestroyTensor(x);
    aclDestroyTensor(groupIndex);
    aclDestroyTensor(out);
    // 7. Release device resources. Modify the code based on the API definition.
    aclrtFree(xDeviceAddr);
    aclrtFree(groupIndexDeviceAddr);
    aclrtFree(outDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}