[object Object]

[object Object][object Object]undefined[object Object]

• API function: Computes MoeFFN and FFN. This operator is a feed-forward network (FFN) operator when there is no expert group ([object Object] being null) and an MoeFFN operator when there is an expert group. Both operators are variants of FFN and use the Mixture-of-Experts (MoE) architecture. MoE is a technology used to train models with trillions of parameters. MoE divides a prediction modeling task into several subtasks, trains an expert model on each subtask, and develops a gating model. The model assigns one or more experts based on the input data, and finally combines the computation results of multiple experts as the prediction result. In the MoE model, the input data is allocated to one or more most relevant experts, and the final result is determined based on the computation results of all involved experts.

  Compared with the API, the expertTokens input of this API is changed from an array to a tensor.

  Compared with the API, this API supports the expertTokens index input, which is distinguished by tokensIndexFlag. The expertTokens input is changed from an array to a tensor.

• Formula:

  • Non-quantization scenario:

    $$y=activation(x * W1 + b1) * W2 + b2$$

  • Quantization scenario:

    $$y=((activation((x * W1 + b1) * deqScale1) * scale + offset) * W2 + b2) * deqScale2$$

  • Pseudo-quantization scenario:

    $$y=activation(x * ((W1 + antiquantOffset1) * antiquantScale1) + b1) * ((W2 + antiquantOffset2) * antiquantScale2) + b2$$

  Note:

  Whether FFN has performance benefits in the scenario without experts or in the scenario with a single expert depends on the actual test situation. When the vector time of the small operator corresponding to the FFN structure on the entire network is more than 30 μs and accounts for more than 10% of the FFN structure, try to use this fusion operator. If the actual test performance deteriorates, do not use this function.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

Description: The variables involved in the parameters are described as follows. M: number of tokens, corresponding to BS (B: batch size of input samples.
S: sequence length of input samples) in Transformer. K1: input channel count of the first MatMul, corresponding to H (head-size, hidden layer size) in Transformer. N1: output channel count of the first MatMul. K2: input channel count of the second MatMul. N2: output channel count of the second MatMul, corresponding to H in Transformer.
[object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: E indicates the number of experts in the expert scenario. G indicates the number of antiquantOffset and antiquantScale groups in the pseudo-quantization per-group scenario.

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computing.

• In all scenarios, K1=N2, K1<65536, K2<65536. The M axis must be less than the maximum value of INT32 after 32-byte alignment.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • The BFLOAT16 data type is supported only by the Atlas 800I A2 inference product.
  • If there are experts, the total number of experts must be the same as M of [object Object].
  • When the activation layer is geglu, swiglu, or reglu, only the FLOAT16 high-performance scenario without expert groups is supported, and N1 = 2 × K2. (The FLOAT16 scenario refers to a scenario where the data type of all required aclTensor parameters is FLOAT16.)
  • When the activation layer is gelu, fastgelu, relu, or silu, FLOAT16 high-precision and high-performance scenarios, BFLOAT16 scenarios, quantization scenarios, and fake-quantization scenarios with experts or without expert groups are supported, and N1=K2.
  • In non-quantization scenarios, do not input quantization or pseudo-quantization parameters. In quantization scenarios, do not input pseudo-quantization parameters. In pseudo-quantization scenarios, do not input quantization parameters.
  • Parameter types in quantization scenarios: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are divided into two cases according to the type of [object Object].
    • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64, INT64, and FLOAT32.
    • When the data type of [object Object] is BFLOAT16, [object Object] supports only the BFLOAT16 data type.
    • The data type of [object Object] must be the same as that of [object Object].
  • Parameter types in quantization scenarios where the per-channel mode of [object Object] is supported: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are categorized into two cases according to the type of [object Object].
    • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64 and INT64.
    • When the data type of [object Object] is BFLOAT16, [object Object] supports only the BFLOAT16 data type.
    • The data type of [object Object] must be the same as that of [object Object].
  • The pseudo-quantization scenario supports two parameter types:
    • The data type of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, and that of [object Object] is FLOAT16. [object Object] supports data types INT8 and INT4.
    • The data type of [object Object] is BFLOAT16, that of [object Object] is BFLOAT16, that of [object Object] is FLOAT32, that of [object Object] is BFLOAT16, and that of [object Object] is BFLOAT16. [object Object] supports data types INT8 and INT4.
  • When the data type of [object Object] or [object Object] is INT4, the last dimension of the shape must be an even number.
  • In pseudo-quantization scenarios, under per-group mode, G (number of groups) in [object Object] and [object Object] must be exactly divided by K1, and G (number of groups) in [object Object] and [object Object] must be exactly divided by K2.
  • In the BFLOAT16 non-quantization scenario, [object Object] can only be set to [object Object]. In the FLOAT16 non-quantization scenario, [object Object] can be set to [object Object] or [object Object]. In quantization or fake-quantization scenarios, [object Object] can be set to [object Object] or [object Object], but the setting does not take effect.
  • If [object Object] is set to [object Object] and there are experts ([object Object] not being null), the value of [object Object] must meet the following requirements: If both [object Object] and [object Object] are valid array indexes in [object Object], and [object Object] is greater than [object Object], then the value of a jth element in [object Object] is greater than or equal to the value of an ith element in [object Object].

• [object Object]Atlas inference accelerator cards[object Object]:

  • Only the non-expert scenario is supported.
  • N1 must be equal to K2.

[object Object]

This fusion operator API supports only the aclnn single-operator calling mode, which is showcased in the following example. For details, see .