AntiQuantize

Applicability

Function Usage

Performs fake quantization by element. For example, apply fake quantization to convert the int8_t type to the half type. This API supports input of data no more than two dimensions.

AntiQuantize has similar function to AscendAntiQuant and is recommended because it has a unified form in different quantization scenarios.

The fake quantization policies of this API include PER_TENSOR, PER_CHANNEL, PER_TOKEN, and PER_GROUP. The dequantization coefficients scale and offset are scalars in the PER_TENSOR scenario and vectors in other scenarios. The formulas are as follows:

• PER_TENSOR scenario (quantization by tensor): The shape of scale and offset is [1].

  

• PER_CHANNEL scenario (quantization by channel): The shape of srcTensor is [m, n]. Each channel dimension corresponds to a quantization parameter. The shape of scale and offset is [1, n].

  

• PER_TOKEN scenario (quantization by token): The elements in each group of tokens (m groups of tokens in the n direction) of srcTensor share a quantization parameter. When the shape of srcTensor is [m, n], the shape of scale and offset is [m, 1].

  

• PER_GROUP scenario (quantization by group): The compute direction of groups is defined as k. In the k direction, each groupSize elements of srcTensor shares a group of scale and offset. When the shape of srcTensor is [m, n], if kDim is 0, k is in the m direction, and the shape of scale and offset is [(m + groupSize – 1)/groupSize, n]; if kDim is 1, k is in the n direction, and the shape of scale and offset is [m, (n + groupSize – 1)/groupSize].

  Based on the input data type, PER_GROUP is classified into two scenarios: fp4x2_e2m1_t/fp4x2_e1m2_t scenario (referred to as the float4 scenario) and int8_t/hifloat8_t/fp8_e5m2_t/fp8_e4m3fn_t scenario (referred to as the b8 scenario).

  • fp4x2_e2m1_t/fp4x2_e1m2_t scenario (float4 scenario)
    • k is the m direction. That means, the i axis in the formula is the compute direction of group (kDim = 0):

      

    • k is the n direction. That means, the j axis in the formula is the compute direction of group (kDim = 1):

      

  • int8_t/hifloat8_t/fp8_e5m2_t/fp8_e4m3fn_t scenario (b8 scenario)
    • k is the m direction. That means, the i axis in the formula is the compute direction of group (kDim = 0):

      

    • k is the n direction. That means, the j axis in the formula is the compute direction of group (kDim = 1):

      

Prototype

• Pass the temporary space through the sharedTmpBuffer input parameter.

  1 2

  template <const AntiQuantizeConfig& config, typename DstT, typename SrcT, typename ScaleT, typename OffsetT> __aicore__ inline void AntiQuantize(const LocalTensor<DstT>& dstTensor, const LocalTensor<SrcT>& srcTensor, const ScaleT& scale, const OffsetT& offset, const LocalTensor<uint8_t>& sharedTmpBuffer, const AntiQuantizeParams& params)

• Allocate the temporary space through the API framework.

  1 2	template <const AntiQuantizeConfig& config, typename DstT, typename SrcT, typename ScaleT, typename OffsetT> __aicore__ inline void AntiQuantize(const LocalTensor<DstT>& dstTensor, const LocalTensor<SrcT>& srcTensor, const ScaleT& scale, const OffsetT& offset, const AntiQuantizeParams& params)

Due to the complex mathematical computation involved in the internal implementation of this API, extra temporary space is required to store intermediate variables generated during computation. The temporary space can be allocated through the API framework or passed by developers through the sharedTmpBuffer input parameter.

• When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the temporary space.

• When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization.

If the API framework is used, developers must reserve the temporary space. If sharedTmpBuffer is used, developers must allocate space for sharedTmpBuffer. To obtain the size of the temporary space (BufferSize) to be reserved, use the API provided in GetAntiQuantizeMaxMinTmpSize.

Parameters

1
2
3
4
5

struct AntiQuantizeConfig {
    AntiQuantizePolicy policy;
    bool hasOffset;
    int32_t kDim = 1;
 }

1
2
3
4
5

struct AntiQuantizeParams {
        uint32_t m;
        uint32_t n;
        uint32_t groupSize = 0;
}

Returns

None

Constraints

• The source operand address must not overlap the destination operand address.
• For details about the operand address alignment requirements, see General Address Alignment Restrictions.
• The length of the data involved in computation of the input and output operands must be 32-byte aligned.
• The data volume in the continuous computation direction (n direction) must be 32-byte aligned.
• In the PER_GROUP float4 scenario, offset is not supported. In this scenario, the hasOffset parameter in the template parameter config must be set to false.

Examples

• PER_CHANNEL, PER_TOKEN, and PER_GROUP modes

  1 2 3 4 5 6 7 8 9 10 11 12

  constexpr static AntiQuantizePolicy tokenPolicy = AntiQuantizePolicy::PER_TOKEN; constexpr static AntiQuantizePolicy channelPolicy = AntiQuantizePolicy::PER_CHANNEL; constexpr static AntiQuantizePolicy groupPolicy = AntiQuantizePolicy::PER_GROUP; // The PER_TOKEN mode is used as an example. Enable the offset function. kDim is valid only in the PER_GROUP scenario, indicating that the compute direction of groups is n. constexpr static AntiQuantizeConfig config = {tokenPolicy, true, 1}; AntiQuantizeParams params; // m and n are external input parameters, indicating the number of elements in the m and n directions involved in srcLocal. params.m = m; params.n = n; params.groupSize = n; // This is valid only in the PER_GROUP scenario. It indicates that all elements in the n direction share a group of scale and offset. // srcLocal is a LocalTensor of the int8_t type, and dstLocal, scale, and offset are a LocalTensor of the half type. AntiQuantize<config>(dstLocal, srcLocal, scale, offset, params);

• PER_TENSOR mode

  1 2 3 4 5 6 7 8 9 10

  constexpr static AntiQuantizePolicy tensorPolicy = AntiQuantizePolicy::PER_TENSOR; // Enable the offset function. constexpr static AntiQuantizeConfig config = {tensorPolicy, true, -1}; AntiQuantizeParams params; // m and n are external input parameters, indicating the number of elements in the m and n directions involved in srcLocal. params.m = m; params.n = n; params.groupSize = 0; // This is valid only in the PER_GROUP scenario. // srcLocal is a LocalTensor of the int8_t type, dstLocal is a LocalTensor of the half type, and scale and offset are scalars of the half type. AntiQuantize<config>(dstLocal, srcLocal, scale, offset, params);

Result example:

Input (srcLocal):
[-4, 2, -2, -3, -1, -4, 1, 3, 4, 1, -2, 0, ... 1]
Input (scale vector):
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 1]
Input (scale scalar):
[1]
Input (offset vector):
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ... 0]
Input (offset scalar):
[0]
Output (dstLocal). In this case, dstLocal = srcLocal:
[-4, 2, -2, -3, -1, -4, 1, 3, 4, 1, -2, 0, ... 1]