SetQuantScalar

Applicability

Function Usage

Quantizes or dequantizes all values in the output matrix using the same coefficient. That is, the entire C matrix corresponds to one quantization parameter, and the shape of the quantization parameter is [1]. For details about quantization and dequantization, see Quantization Scenarios.

Matmul dequantization scenario: During Matmul computation, the input of the left and right matrices is of the int8_t or int4b_t type, and the output is of the half type. Alternatively, both the input and output of the left and right matrices are of the int8_t type. In this scenario, when the output matrix C is moved from CO1 to the global memory, dequantization is performed to dequantize the final result to the corresponding half or int8_t type.

Matmul quantization scenario: During Matmul computation, the input of the left and right matrices is of the half or bfloat16_t type, and the output is of the int8_t type. In this scenario, when the output matrix C is moved from CO1 to the global memory, quantization is performed to quantize the final result to the int8_t type.

Prototype

1

__aicore__ inline void SetQuantScalar(const uint64_t quantScalar)

Parameters

The formula for converting the quantization parameters scale and offset of the float type to the input parameters of the uint64 type is as follows:

1. quantScalar is in 64-bit format and is initialized to 0.
2. The higher 19 bits of scale are truncated and stored in bit 32 of quantScalar, and bit 46 is changed to 1.

   quantScalar = quantScalar∣ (scale & 0xFFFFE000) ∣ (1 << 46)

3. The subsequent computation is performed based on the value of offset.
   • If offset does not exist, no subsequent computation is performed.
   • If offset exists:
     1. The value of offset is converted to an int value in the range of [–256, 255].

        offset = Max(Min(INT(Round(offset)), 255), −256)

     2. Nine bits of offset are retained and stored in bits 37 to 45 of quantScalar.

        quantScalar = (quantScalar & 0x4000FFFFFFFF) ∣ ((offset & 0x1FF) << 37)

Returns

None

Restrictions

The value must be the same as that of SetDequantType.

This API must be called before Iterate or IterateAll.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm, &tiling);
float tmp = 0.1;  // Multiplied by 0.1 during GM output
// Convert the quantization or dequantization coefficient of the floating-point value to the uint64_t type.
uint64_t ans = static_cast<uint64_t>(*reinterpret_cast<int32_t*>(&tmp));
mm.SetQuantScalar(ans);
mm.SetTensorA(gm_a);
mm.SetTensorB(gm_b);
if (tiling.isBias) {
    mm.SetBias(biasGlobal);
}
mm.IterateAll(gm_c);
mm.End();