[object Object][object Object][object Object]undefined[object Object]

• Description: Quantizes the input x, performs grouped matrix multiplication, and dequantizes the result.
• Formula:
  1. If smoothScaleOptional is input, then:
  $$x = x\cdot scale_{smooth}$$
  2. If xScaleOptional is not input, dynamic quantization is performed, and the x quantization coefficient needs to be calculated.
  $$scale_{x}=row\_max(abs(x))/max_{quantDataType}$$
  3. Quantization
  $$x_{quantized}=round(x/scale_{x})$$
  4. Grouped matrix multiplication + dequantization
  • 4.1 If the data type of the input $scale_{weight}$ is FLOAT32, then:
  $$\begin{aligned} x^{*}_{quantized} &= x_{quantized}[group[i-1]:group[i]]\\ out^{*}_{quantized} &= out_{quantized}[group[i-1]:group[i]]\\ scale^{*}_{x} &= \begin{cases} scale_{x}[group[i-1]:group[i]] & pertoken \\ scale_{x} & pertensor \\ \end{cases} \\ out^{*}_{quantized} &= (x^{*}_{quantized}@weight_{quantized}[i] + bias) * scale_{weight}[i] * scale^{*}_{x} \end{aligned}$$
  • 4.2 If the data type of the input $scale_{weight}$ is INT64, then:$$x^{*}_{quantized} = x_{quantized}[group[i-1]:group[i]] \\ out^{*}_{quantized} = out_{quantized}[group[i-1]:group[i]] \\ scale_{weight} = torch.tensor(np.frombuffer(scale_{weight}.numpy().astype(np.int32).\\tobytes(), dtype=np.float32)).reshape(scale_{weight}) \\ out^{*}_{quantized} = (x^{*}_{quantized}@weight_{quantized}[i] + bias) * scale_{weight}[i]$$ Note: In the scenario described in 4.2, the matrix multiplication operation has been performed on $scale_{weight}$ and $scale_{x}$ before the input of $scale_{weight}$. Therefore, this step is omitted during internal operator computation. This requires that the scenario must be per-tensor static quantization. That is, before input, $scale_{weight}$ must be processed as follows to obtain data of the INT64 type:$$scale_{weight} = scale_{weight} * scale_{x} \\ scale_{weight} = torch.tensor(np.frombuffer(scale_{weight}.numpy().astype(np.float32). \\tobytes(), dtype=np.int32).astype(np.int64)).reshape(scale_{weight})$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]
  • In ND format, the shape of [object Object] can be three dimensions.
    • When [object Object] is set to [object Object], the dimensions are (g, n, k).
    • When [object Object] is set to [object Object], the dimensions are (g, k, n).
  • In FRACTAL_NZ format, the shape of [object Object] can be five dimensions.
    • When [object Object] is set to true, the dimensions are (g, k1, n1, n0, k0), where k0 = 32, n0 = 16, and k1 and k of [object Object] must satisfy ceilDiv(k, 32) = k1.
    • When [object Object] is set to [object Object], the dimensions are (g, n1, k1, k0, n0), where k0 = 16, n0 = 32, and k1 and k of [object Object] must meet the following relationship: ceilDiv(k, 16) = k1.
    • [object Object] and [object Object] can be used to convert the input format from ND to FRACTAL_NZ.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a deterministic implementation.

• n and k must be integer multiples of 16.

• If the data type of [object Object] is INT64, the data type of [object Object] must be FLOAT16 and the value of [object Object] must be [object Object]. If the data type of [object Object] is FLOAT16, the data type of [object Object] must be INT64 and the value of [object Object] must be [object Object].

[object Object]

The following example is for reference only. For details, see .