AscendDequant

Applicability

Function Usage

Performs dequantization by element. For example, dequantize the int32_t data type to the half or float data type. This API supports input of data no more than two dimensions.

• Assuming that the shape of the input srcTensor is (m, n), the number of bytes occupied by each row of data (n pieces of input data) must be 32-byte aligned, and the number of elements to be dequantized in each row is calCount.
• The dequantization coefficient deqScale can be a scalar or vector. If deqScale is a vector, calCount is less than or equal to the number of elements in deqScale, and only the first CalCount dequantization coefficients take effect.
• The shape of the output dstTensor is (m, n_dst). If n x sizeof(dstT) does not meet the 32-byte alignment requirement, it needs to be padded to 32 bytes. n_dst indicates the number of columns after the padding.

The following provides two specific examples to explain the parameter configurations and compute logics: (The DequantParams type in the following is a {m, n, calCount} structure that stores shape information.)

• As shown in the following figure, the data type of srcTensor is int32_t, m equals 4, n equals 8, and calCount equals 4, indicating that the number of elements to be dequantized in each row of srcTensor is 4, the first four elements in deqScale take effect, and the last 12 elements are not involved in dequantization. The data type of dstTensor is bfloat16_t, m equals 4, and n_dst equals 16 (16 × sizeof(bfloat16_t) % 32 = 0). The compute logic is that every n elements in srcTensor are arranged in a row. For the first calCount elements in each row, the ith element of srcTensor is multiplied by the ith element of deqScale and this product is written into the ith element in the corresponding row of dstTensor. Elements from calCount + 1 to n_dst in the corresponding row of dstTensor are uncertain values.

  

• As shown in the following figure, the data type of srcTensor is int32_t, m equals 4, n equals 8, and calCount equals 4, indicating that the number of elements to be dequantized in each row of srcTensor is 4. The data type of dstTensor is float, m equals 4, and n_dst equals 8 (8 × sizeof(float) % 32 = 0). The first four elements in each row of srcTensor are multiplied by the scalar deqScale and the products are written to the corresponding positions in each row of dstTensor.

  

Set the template parameter mode to DEQUANT_WITH_SINGLE_ROW:

If DequantParams {m, n, calCount} meets the following three conditions:

1. m = 1
2. calCount is a multiple of 32/sizeof (dstT).
3. n % calCount = 0

In this case, {1, n, calCount} is considered as {n/calCount, calCount, calCount} for dequantization.

The following figure shows the effect, in which the passed DequantParams is {1, 16, 8}. Since dstT is float, calCount must be a multiple of 8. In DEQUANT_WITH_SINGLE_ROW mode, {1, 2 × 8, 8} is converted to {2, 8, 8} for calculation.

• PER_TOKEN dequantization: The elements in each group of tokens (m groups of tokens in the n direction) of srcTensor share the deqscale parameter. When srcTensor is [m, n], deqscale is [m, 1].

  

• PER_GROUP dequantization: The compute direction of groups is defined as k. In the k direction, each groupSize elements of srcTensor share the deqscale parameter. When the shape of srcTensor is [m, n], if kDim is 0, k is in the m direction, and the shape of deqscale is [(m + groupSize – 1)/groupSize, n]; if kDim is 1, k is in the n direction, and the shape of deqscale is [m, (n + groupSize – 1)/groupSize].
  • kDim = 0:

    

  • kDim = 1:

    

Principles

The figure below illustrates the internal algorithm block diagram of AscendDequant high-level APIs, taking the input srcTensor with a data type of int32_t and a shape of [m, n], input deqScale with a data type of scaleT and a shape of [n], and output dstTensor with a data type of dstT and a shape of [m, n] as examples.

Figure 1 AscendDequant internal algorithm block diagram

The computation process is divided into the following steps, all of which are performed on vectors:

1. Precision conversion: Convert srcTensor and deqScale into FP32 tensors to obtain srcFP32 and deqScaleFP32, respectively.
2. Mul computation: srcFP32 has a total of m rows, each with a length of n. Each row of srcFP32 is multiplied by deqScaleFP32 through m cycles. The mask limits the Mul computation to only the first dequantParams.calcount elements. The value range of index in the figure is [0, m), corresponding to each row of srcFP32. The calculation result is mulRes, whose shape is [m, n].
3. Precision conversion of result data: Convert mulRes from FP32 to a dstT tensor. The result is dstTensor with a shape of [m, n].

Prototype

• The deqScale parameter is a vector.
  • Pass the temporary space through the sharedTmpBuffer input parameter.

    1 2

    template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale, const LocalTensor<uint8_t>& sharedTmpBuffer, DequantParams params)

  • Allocate the temporary space through the API framework.

    1 2	template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale, DequantParams params)

  • PER_TOKEN or PER_GROUP quantization
    It is supported only by the Atlas 350 Accelerator Card.

    • Pass the temporary space through the sharedTmpBuffer input parameter.

      1 2

      template <typename dstT, typename srcT, typename scaleT, const AscendDeQuantConfig& config, const AscendDeQuantPolicy& policy> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<srcT>& srcTensor, const LocalTensor<scaleT>& scaleTensor, const LocalTensor<scaleT>& offsetTensor, const LocalTensor<uint8_t>& sharedTmpBuffer, const AscendDeQuantParam& para)

    • Allocate the temporary space through the API framework.

      1 2

      template <typename dstT, typename srcT, typename scaleT, const AscendDeQuantConfig& config, const AscendDeQuantPolicy& policy> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<srcT>& srcTensor, const LocalTensor<scaleT>& scaleTensor, const LocalTensor<scaleT>& offsetTensor, const AscendDeQuantParam& para)

• The deqScale parameter is a scalar.
  • Pass the temporary space through the sharedTmpBuffer input parameter.

    1 2

    template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const scaleT deqScale, const LocalTensor<uint8_t>& sharedTmpBuffer, DequantParams params)

  • Allocate the temporary space through the API framework.

    1 2	template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW> __aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const scaleT deqScale, DequantParams params)

Due to the complex mathematical computation involved in the internal implementation of this API, extra temporary space is required to store intermediate variables generated during computation. The temporary space can be allocated through the API framework or passed by developers through the sharedTmpBuffer input parameter.

• When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the temporary space.

• When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization.

If the API framework is used, developers must reserve the temporary space. If sharedTmpBuffer is used, developers must allocate space for sharedTmpBuffer. To obtain the size of the temporary space (BufferSize) to be reserved, use the GetAscendDequantMaxMinTmpSize API provided in GetAscendDequantMaxMinTmpSize.

The following APIs are not recommended. Do not use them for newly developed content:

1
2

template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW>
__aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale, const LocalTensor<uint8_t>& sharedTmpBuffer, const uint32_t calCount)

1
2

template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW>
__aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale, const LocalTensor<uint8_t>& sharedTmpBuffer)

1
2

template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW>
__aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale, const uint32_t calCount)

1
2

template <typename dstT, typename scaleT, DeQuantMode mode = DeQuantMode::DEQUANT_WITH_SINGLE_ROW>
__aicore__ inline void AscendDequant(const LocalTensor<dstT>& dstTensor, const LocalTensor<int32_t>& srcTensor, const LocalTensor<scaleT>& deqScale)

Parameters

1
2
3
4

struct AscendDeQuantConfig {
        bool hasOffset;
        int32_t kDim = 1;
}

1
2
3
4
5
6

enum class AscendDeQuantPolicy : int32_t {
        PER_TOKEN // PER_TOKEN mode
        PER_GROUP // PER_GROUP mode
        PER_CHANNEL_PER_GROUP // Reserved.
        PER_TOKEN_PER_GROUP // Reserved.
}

1
2
3
4
5
6

struct DequantParams
{
    uint32_t m;             // number of rows of srcTensor
    uint32_t n;             // number of columns of srcTensor
    uint32_t calCount;      // For each row of srcTensor, the first calCount elements are valid data, and they are multiplied by the first calCount elements of deqScale or by the deqScale scalar.
};

1
2
3
4
5
6

struct AscendDeQuantParam {
        uint32_t m;
        uint32_t n;
        uint32_t calCount;   
        uint32_t groupSize = 0;
}

Returns

None

Constraints

• The source operand address must not overlap the destination operand address.
• For details about the operand address alignment requirements, see General Address Alignment Restrictions.
• Currently, only the Atlas 350 Accelerator Card supports the PER_TOKEN and PER_GROUP scenarios.
• In the PER_TOKEN and PER_GROUP scenarios, the size of data in the continuous compute direction (n direction) must be 32-byte aligned.

Examples

For a complete operator sample, see dequant operator sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

// dstLocal: tensor for storing the dequantization computation result
// srcLocal: tensor for storing the dequantization computation input
// deqScaleLocal: tensor for storing the input dequantization coefficient for dequantization computation
// sharedTmpBuffer: tensor for storing the temporary buffer during dequantization computation

uint32_t m = 4; // number of rows of srcTensor
uint32_t n = 8; // number of columns of srcTensor
uint32_t calCount = 6; // the first calCount elements in each row of srcTensor involved in dequantization computation

// deqScale is a vector, whose temporary space is passed through the sharedTmpBuffer input parameter.
AscendC::AscendDequant(dstLocal, srcLocal, deqScaleLocal, sharedTmpBuffer, {m, n, calCount});

// deqScale is a vector, whose temporary space is allocated through the API framework.
AscendC::AscendDequant(dstLocal, srcLocal, deqScaleLocal, {m, n, calCount});

// deqScale is a scalar, whose temporary space is passed through the sharedTmpBuffer input parameter.
AscendC::AscendDequant(dstLocal, srcLocal, static_cast<float>(2.2), sharedTmpBuffer, {m, n, calCount});
// deqScale is a scalar, whose temporary space is allocated through the API framework.
AscendC::AscendDequant(dstLocal, srcLocal, static_cast<float>(2.2), {m, n, calCount});

Result example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Input (srcLocal) of the int32_t data type:
[[-16 -15 -14 -13 -12 -11 -10  -9]
 [ -8  -7  -6  -5  -4  -3  -2  -1]
 [  0   1   2   3   4   5   6   7]
 [  8   9  10  11  12  13  14  15]]

deqScaleLocal of the float data type:
[2.2  -2.2  2.2  -2.2  2.2  -2.2  0.  0.]

Output (dstLocal) of the float data type:
[[-35.2  33.  -30.8  28.6 -26.4  24.2 -10.   -9. ]
 [-17.6  15.4 -13.2  11.   -8.8   6.6  -2.   -1. ]
 [  0.   -2.2   4.4  -6.6   8.8 -11.    6.    7. ]
 [ 17.6 -19.8  22.  -24.2  26.4 -28.6  14.   15. ]]

The following is an example of calling the API in the PER_TOKEN and PER_GROUP scenarios:

1
2
3
4
5
6
7
8
9

// Note that m and n must be passed from an external system.
constexpr static bool isReuseSource = false;
constexpr static AscendDeQuantConfig config = {has_offset, -1};
constexpr static AscendDeQuantPolicy policy = AscendDeQuantPolicy::PER_TOKEN; // The enumerated value can be modified to enable PER_GROUP.
AscendDeQuantParam para;
para.m = m;
para.n = n;
para.calCount = calCount;
AscendDequant<dstType, srcType, scaleType, config, policy>(dstLocal, srcLocal, scaleLocal, offsetLocal, para);