[object Object]

[object Object][object Object]undefined[object Object]

• Function: This fused operator performs two small-matrix multiplications on the input matrix x at a time. That is, it performs right multiplication on the input matrix kroneckerP2 and left multiplication on the input matrix kroneckerP1, and then performs quantization on the matrix multiplication result. Currently, the pertoken and pergroup quantization modes are supported, which correspond to the int4 and float4_e2m1 quantization output types, respectively.

• The matrix multiplication formula is as follows:

  1. Right-multiply x by kroneckerP2:
  $$x' = x @ kroneckerP2$$
  2. Left-multiply x' by kroneckerP1:
  $$x'' = kroneckerP1@x'$$

• Quantization calculation method:

  Pertoken quantization mode:

  1. Calculate the maximum absolute value along dimension 0 of x'' and divide the value by (7/clipRatio) to calculate the quantization factor to be quantized to the int4 format.
  $$quantScale = [max(abs(x''[0,:,:])),max(abs(x''[1,:,:])),...,max(abs(x''[K,:,:]))]/(7 / clipRatio)$$
  2. Calculate the output:
  $$out = x'' / quantScale$$

  Per-group quantization mode:

  1. After the matrix multiplication, the shape of x'' is [K, M, N]. When calculating mx_quantize in per-group quantization mode, reshape x'' to [K, M*N], which is denoted as x2.

  2. Group elements in the second dimension of x2 by groupsize, including elements e0, e1,..., e31. Calculate emax.

  $$emax = max(e0,e1....e31)$$
  3. Calculate reduceMaxValue and sharedExp.
  $$reduceMaxValue = log2(reduceMax(x2),groupSize=32)$$ $$sharedExp[K,M*N/32] = reduceMaxValue -emax$$
  4. Calculate quantScale.
  $$quantScale[K,M*N/32] = 2 ^ {sharedExp[K,M*N/32]}$$
  5. Each groupsize shares one quantScale and calculates out.
  $$out = x2 / quantScale$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • aclnnFlatQuant defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .