SetQuantVector

Applicability

Function Usage

Quantizes or dequantizes the output matrix using a vector. The input parameter is a vector with shape [1, N], where the value of N is the same as that in M/N/K during Matmul matrix computation. For each column of the output matrix, the coefficient from the corresponding element of the vector is applied to perform quantization or dequantization. For details about quantization and dequantization, see Quantization Scenarios.

Matmul dequantization scenario: During Matmul computation, the input of the left and right matrices is of the int8_t or int4b_t type, and the output is of the half type. Alternatively, both the input and output of the left and right matrices are of the int8_t type. In this scenario, when the output matrix C is moved from CO1 to the global memory, dequantization is performed to dequantize the final result to the corresponding half or int8_t type.

Matmul quantization scenario: During Matmul computation, the input of the left and right matrices is of the half or bfloat16_t type, and the output is of the int8_t type. In this scenario, when the output matrix C is moved from CO1 to the global memory, quantization is performed to quantize the final result to the int8_t type.

Prototype

• Quantization parameters stored in the GM

  1	__aicore__ inline void SetQuantVector(const GlobalTensor<uint64_t>& quantTensor)

• Quantization parameters stored in the L1 Buffer

  1	__aicore__ inline void SetQuantVector(const LocalTensor<uint64_t>& quantTensor)

  • For the Atlas 200I/500 A2 inference product , quantization parameters cannot be stored in the L1 Buffer.
  • For the Atlas inference product AI Core, quantization parameters cannot be stored in the L1 Buffer.

Parameters

The formula for converting the quantization parameters scale and offset of the float type to the input parameters of the uint64 type is as follows:

1. quantTensor is in 64-bit format and is initialized to 0.
2. The higher 19 bits of scale are truncated and stored in bit 32 of quantTensor, and bit 46 is changed to 1.

   quantTensor = quantTensor ∣ (scale & 0xFFFFE000) ∣ (1 ≪ 46)

3. The subsequent computation is performed based on the value of offset.
   • If offset does not exist, no subsequent computation is performed.
   • If offset exists:
     1. The value of offset is converted to an int value in the range of [–256, 255].

        offset = Max(Min(INT(Round(offset)), 255), −256)

     2. Nine bits of offset are retained and stored in bits 37 to 45 of quantTensor.

        quantTensor = (quantTensor & 0x4000FFFFFFFF) ∣ ((offset & 0x1FF) ≪ 37)

Returns

None

Restrictions

The value must be the same as that of SetDequantType.

This API must be called before Iterate or IterateAll.

Example

For details about a complete example, see matmul_quant sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm, &tiling);

// quantTensor: tensor used during quantization or dequantization operations. The supported storage locations are GM and L1.
mm.SetQuantVector(quantTensor);

mm.SetTensorA(gm_a);
mm.SetTensorB(gm_b);
if (tiling.isBias) {
    matmulObj.SetBias(biasGlobal);
}
mm.IterateAll(gm_c);
mm.End();