[object Object]

[object Object][object Object]undefined[object Object]

• API function: Computes MoeFFN and FFN. This operator is a feed-forward network (FFN) operator when there is no expert group ([object Object] being null) and an MoeFFN operator when there is an expert group. Both operators are variants of FFN and use the Mixture-of-Experts (MoE) architecture. MoE is a technology used to train models with trillions of parameters. MoE divides a prediction modeling task into several subtasks, trains an expert model on each subtask, and develops a gating model. The model assigns one or more experts based on the input data, and finally combines the computation results of multiple experts as the prediction result. In the MoE model, the input data is allocated to one or more most relevant experts, and the final result is determined based on the computation results of all involved experts.

• Formula:

  • Non-quantization scenario:

    $$y=activation(x * W1 + b1) * W2 + b2$$

  • Quantization scenario:

    $$y=((activation((x * W1 + b1) * deqScale1) * scale + offset) * W2 + b2) * deqScale2$$

  • Pseudo-quantization scenario:

    $$y=activation(x * ((W1 + antiquantOffset1) * antiquantScale1) + b1) * ((W2 + antiquantOffset2) * antiquantScale2) + b2$$

Note:

Whether FFN has performance benefits in the scenario without experts or in the scenario with a single expert depends on the actual test situation. When the vector time of the small operator corresponding to the FFN structure on the entire network is more than 30 μs and accounts for more than 10% of the FFN structure, try to use this fusion operator. If the actual test performance deteriorates, do not use this function.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

Note: M indicates the number of tokens, which corresponds to the batch size (BS) in Transformer. B indicates the batch size, and S indicates the sequence length. K1 indicates the number of input channels of the first matmul, which corresponds to the head size (H) in Transformer. N1 indicates the number of output channels of the first matmul, and K2 indicates the number of input channels of the second matmul. N2 indicates the number of output channels of the second matmul, which corresponds to the head size (H) in Transformer. E indicates the number of experts in the expert scenario. G indicates the number of groups of antiquantOffset and antiquantScale in the fake-quantization per-group scenario.

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:
  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computing.
• If there are experts, the total number of experts must be the same as M of [object Object].
• When the activation layer is geglu, swiglu, or reglu, only the FLOAT16 high-performance scenario without expert groups is supported, and N1 = 2 × K2. (The FLOAT16 scenario refers to a scenario where the data type of all required aclTensor parameters is FLOAT16.)
• When the activation layer is gelu, fastgelu, relu, or silu, FLOAT16 high-precision and high-performance scenarios, BFLOAT16 scenarios, quantization scenarios, and fake-quantization scenarios with experts or without expert groups are supported, and N1=K2.
• In all scenarios, K1=N2, K1<65536, K2<65536. The M axis must be less than the maximum value of INT32 after 32-byte alignment.
• In non-quantization scenarios, do not input quantization or pseudo-quantization parameters. In quantization scenarios, do not input pseudo-quantization parameters. In pseudo-quantization scenarios, do not input quantization parameters.
• Parameter types in quantization scenarios: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are divided into two cases according to the type of [object Object].
  • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64, INT64, and FLOAT32.
  • When the data type of [object Object] is BFLOAT16, [object Object] supports the BFLOAT16 data type.
  • The data types of [object Object] and [object Object] must be consistent.
• Parameter types in quantization scenarios where the per-channel mode of [object Object] is supported: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are categorized into two cases according to the type of [object Object].
  • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64 and INT64.
  • When the data type of [object Object] is BFLOAT16, [object Object] supports the BFLOAT16 data type.
  • The data types of [object Object] and [object Object] must be consistent.
• The pseudo-quantization scenario supports two parameter types:
  • The data type of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, and that of [object Object] is FLOAT16. [object Object] supports data types INT8 and INT4.
  • The data type of [object Object] is BFLOAT16, that of [object Object] is BFLOAT16, that of [object Object] is FLOAT32, that of [object Object] is BFLOAT16, and that of [object Object] is BFLOAT16. [object Object] supports data types INT8 and INT4.
• When the data type of [object Object] or [object Object] is INT4, the last dimension of the shape must be an even number.
• In pseudo-quantization scenarios, under per-group mode, K1 in [object Object] and [object Object] must be exactly divided by G (number of groups), and K2 in [object Object] and [object Object] must be exactly divided by G (number of groups).
• In pseudo-quantization scenarios, under per-group mode, the data type of [object Object] can only be INT4.
• In the BFLOAT16 non-quantization scenario, [object Object] can only be set to [object Object]. In the FLOAT16 non-quantization scenario, [object Object] can be set to [object Object] or [object Object]. In quantization or fake-quantization scenarios, [object Object] can be set to [object Object] or [object Object], but the setting does not take effect.

[object Object]

The following example is for reference only. For details, see .