Dequantize

Applicability

Function Usage

Performs dequantization by element. For example, dequantize the int32_t data type to the half or float data type. This API supports input of data no more than two dimensions.

Dequantize has similar function to AscendDequant and is recommended because it has a unified form in different quantization scenarios.

Prototype

• Pass the temporary space through the sharedTmpBuffer input parameter.

  1 2

  template <const DequantizeConfig& config, typename DstT, typename SrcT, typename ScaleT, typename OffsetT> __aicore__ inline void Dequantize(const LocalTensor<DstT>& dstTensor, const LocalTensor<SrcT>& srcTensor, const ScaleT& scale, const OffsetT& offset, const LocalTensor<uint8_t>& sharedTmpBuffer, const DequantizeParams& params)

• Allocate the temporary space through the API framework.

  1 2	template <const DequantizeConfig& config, typename DstT, typename SrcT, typename ScaleT, typename OffsetT> __aicore__ inline void Dequantize(const LocalTensor<DstT>& dstTensor, const LocalTensor<SrcT>& srcTensor, const ScaleT& scale, const OffsetT& offset, const DequantizeParams& params)

  Due to the complex mathematical computation involved in the internal implementation of this API, extra temporary space is required to store intermediate variables generated during computation. The temporary space can be allocated through the API framework or passed by developers through the sharedTmpBuffer input parameter.

  • When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the temporary space.
  • When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization.

  If the API framework is used, developers must reserve the temporary space. If sharedTmpBuffer is used, developers must allocate space for sharedTmpBuffer. To obtain the size of the temporary space (BufferSize) to be reserved, use the GetDequantizeMaxMinTmpSize API provided in GetDequantizeMaxMinTmpSize.

Parameters

1
2
3
4
5

struct DequantizeConfig {
    DequantizePolicy policy;
    bool hasOffset = false;
    int32_t kDim = 1;
}

1
2
3
4
5

struct DequantizeParams {
        uint32_t m;
        uint32_t n;
        uint32_t groupSize = 0;
}

Returns

None

Constraints

• The source operand address must not overlap the destination operand address.
• For details about the operand address alignment requirements, see General Address Alignment Restrictions.
• The data volume in the continuous computation direction (n direction) must be 32-byte aligned.

Examples

• PER_CHANNEL, PER_TOKEN, and PER_GROUP modes

  1 2 3 4 5 6 7 8 9 10 11 12

  constexpr static DequantizePolicy tokenPolicy = DequantizePolicy::PER_TOKEN; constexpr static DequantizePolicy channelPolicy = DequantizePolicy::PER_CHANNEL; constexpr static DequantizePolicy groupPolicy = DequantizePolicy::PER_GROUP; // The PER_CHANNEL mode is used as an example. Disable the offset function. kDim is valid only in the PER_GROUP scenario, indicating that the compute direction of groups is n. constexpr static DequantizeConfig config = {channelPolicy, false, 1}; DequantizeParams params; // m and n are external input parameters, indicating the number of elements in the m and n directions involved in srcLocal. params.m = m; params.n = n; params.groupSize = n; // This is valid only in the PER_GROUP scenario. It indicates that all elements in the n direction share a group of scale and offset. // srcLocal is a LocalTensor of the int32_t type, and dstLocal, scaleLocal, and offsetLocal are a LocalTensor of the float type. Dequantize<config>(dstLocal, srcLocal, scaleLocal, offsetLocal, params); // offsetLocal is a reserved parameter and can be set to an empty tensor.

• PER_TENSOR mode

  1 2 3 4 5 6 7 8 9 10

  constexpr static DequantizePolicy tensorPolicy = DequantizePolicy::PER_TENSOR; // Disable the offset function. constexpr static DequantizeConfig config = {tensorPolicy, false, -1}; DequantizeParams params; // m and n are external input parameters, indicating the number of elements in the m and n directions involved in srcLocal. params.m = m; params.n = n; params.groupSize = 0; // This is valid only in the PER_GROUP scenario. // srcLocal is a LocalTensor of the int32_t type, dstLocal is a LocalTensor of the float type, and scale and offset are scalars of the float type. Dequantize<config>(dstLocal, srcLocal, scale, offset, params); // offset is a reserved parameter and can be set to 0.

Result example:

Input (srcLocal):
[-4, 2, -2, -3, -1, -4, 1, 3, 4, 1, -2, 0, ... 1]
Input (scale vector):
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 1]
Input (scale scalar):
[1]
Output (dstLocal). In this case, dstLocal = srcLocal:
[-4, 2, -2, -3, -1, -4, 1, 3, 4, 1, -2, 0, ... 1]