[object Object][object Object][object Object]undefined[object Object]

• API function: Computes MoeFFN and FFN. This operator is a feed-forward network (FFN) operator when there is no expert group ([object Object] being null) and an MoeFFN operator when there is an expert group. Both operators are variants of FFN and use the Mixture-of-Experts (MoE) architecture. MoE is a technology used to train models with trillions of parameters. MoE divides a prediction modeling task into several subtasks, trains an expert model on each subtask, and develops a gating model. The model assigns one or more experts based on the input data, and finally combines the computation results of multiple experts as the prediction result. In the MoE model, the input data is allocated to one or more most relevant experts, and the final result is determined based on the computation results of all involved experts. Compared with the API, [object Object] in the API is changed from an array to tensor input. Compared with the API, the API supports the [object Object] index input, which is distinguished by [object Object]. The input of [object Object] is changed from an array to tensor input.

• Formula:

  • Non-quantization scenario:

    $$y=activation(x * W1 + b1) * W2 + b2$$

  • Quantization scenario:

    $$y=((activation((x * W1 + b1) * deqScale1) * scale + offset) * W2 + b2) * deqScale2$$

  • Pseudo-quantization scenario:

    $$y=activation(x * ((W1 + antiquantOffset1) * antiquantScale1) + b1) * ((W2 + antiquantOffset2) * antiquantScale2) + b2$$

  Note: Whether FFN has performance benefits in the scenario without experts or in the scenario with a single expert depends on the actual test situation. When the vector time of the small operator corresponding to the FFN structure on the entire network is more than 30 μs and accounts for more than 10% of the FFN structure, try to use this fusion operator. If the actual test performance deteriorates, do not use this function.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

Description: The variables involved in the parameters are described as follows.

• Parameters

  • [object Object] (aclTensor*, compute input): required parameter, aclTensor on the device, x in the formula. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, or INT8. The supported input dimensions range from two dimensions [M, K1] to eight dimensions.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16. The supported input dimensions are two dimensions [M, K1].

  • [object Object] (aclTensor*, compute input): required parameter, aclTensor on the device, expert weight data, W1 in the formula. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, INT8, or INT4. The input is [E, K1, N1] with experts, or [K1, N1] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16. The supported input dimensions are two dimensions [K1, N1].

  • [object Object] (aclTensor*, compute input): required parameter, aclTensor on the device, expert weight data, W2 in the formula. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, INT8, or INT4. The input is [E, K2, N2] with experts, or [K2, N2] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16. The supported input dimensions are two dimensions [K2, N2].

  • [object Object] (aclTensor*, compute input): optional parameter, aclIntArray on the device, representing the number of tokens for each expert. The data type can be INT64. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The maximum length is 256 when the parameter is not null.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, weight data correction value, b1 in the formula. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, FLOAT32, or INT32. The input is [E, N1] with experts, or [N1] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16. The supported input dimension is one dimension [N1].

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, weight data correction value, b2 in the formula. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16, FLOAT32, or INT32. The input is [E, N2] with experts, or [N2] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16. The supported input dimension is one dimension [N2].

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, quantization parameter, quantization scaling factor. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT32. In per-tensor mode, the input is [E] or [1], with or without experts, respectively. In per-channel mode, the input is [E, N1] or [N1], with or without experts, respectively.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, quantization parameter, quantization offset. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be a one-dimensional FLOAT32 vector. The number of input elements is [E] with experts or [1] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, quantization parameter, dequantization scaling factor for the first MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be UINT64, INT64, FLOAT32, or BFLOAT16. The input is [E, N1] with experts, or [N1] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, quantization parameter, dequantization scaling factor for the second MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be UINT64, INT64, FLOAT32, or BFLOAT16. The input is [E, N2] with experts, or [N2] without experts.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, pseudo-quantization parameter, scaling factor for the first MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16. In per-channel mode, the input is [E, N1] or [N1], with or without experts, respectively. In per-group mode, the input is [E, G, N1] or [G, N1], with or without experts, respectively.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, pseudo-quantization parameter, scaling factor for the second MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16. In per-channel mode, the input is [E, N2] or [N2], with or without experts, respectively. In per-group mode, the input is [E, G, N2] or [G, N2], with or without experts, respectively.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, pseudo-quantization parameter, offset for the first MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16. In per-channel mode, the input is [E, N1] or [N1], with or without experts, respectively. In per-group mode, the input is [E, G, N1] or [G, N1], with or without experts, respectively.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (aclTensor*, compute input): optional parameter, aclTensor on the device, pseudo-quantization parameter, offset for the second MatMul. The can be ND.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16. In per-channel mode, the input is [E, N2] or [N2], with or without experts, respectively. In per-group mode, the input is [E, G, N2] or [G, N2], with or without experts, respectively.
    • [object Object]Atlas inference accelerator cards[object Object]: Only null pointers can be passed.

  • [object Object] (char*, compute input): required parameter, attribute value on the host, representing the activation function used, activation in the formula.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: fastgelu, gelu, relu, silu,geglu, swiglu, and reglu are supported.
    • [object Object]Atlas inference accelerator cards[object Object]: fastgelu, gelu, relu, and silu are supported.

  • [object Object] (int64_t, compute input): optional parameter, int on the host, indicating the high-precision or high-performance mode. The data type is INT64.

    • If [object Object] is set to 0, the high-precision mode is enabled. In non-quantization scenarios where all mandatory parameters are FLOAT16, the input and output of the activation layer within the operator are calculated using the FLOAT32 data type.
    • If [object Object] is set to 1, the high-performance mode is used.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This parameter only takes effect when all mandatory parameters are FLOAT16 in non-quantization scenarios. High-precision and high-performance modes are not distinguished in other scenarios.
    • [object Object]Atlas inference accelerator cards[object Object]: The value can only be 1.

  • [object Object] (bool, compute input): optional parameter, bool on the host, indicating whether expertTokens is an index value. The data type can be bool.

    • If [object Object] is set to [object Object], [object Object] is an index value.
    • If [object Object] is set to [object Object], [object Object] is the number of tokens of each expert.

  • [object Object] (aclTensor*, compute output): aclTensor on the device, output y in the formula. The can be ND. The output shape is the same as that of [object Object].

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be FLOAT16.

  • workspaceSize (uint64_t*, output): size of the workspace to be allocated on the device.

  • [object Object] (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns

  aclnnStatus status code. For details, see .

  The first-phase API call completes input parameter verification. The possible error codes and causes are as follows:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  aclnnStatus status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computing.

• In all scenarios, K1=N2, K1<65536, K2<65536. The M axis must be less than the maximum value of INT32 after 32-byte alignment.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • The BFLOAT16 data type is supported only by [object Object]Atlas A2 inference products[object Object].
  • If there are experts, the total number of experts must be the same as M of [object Object].
  • When the activation layer is geglu, swiglu, or reglu, only the FLOAT16 high-performance scenario without expert groups is supported, and N1 = 2 × K2. (The FLOAT16 scenario refers to a scenario where the data type of all required aclTensor parameters is FLOAT16.)
  • When the activation layer is gelu, fastgelu, relu, or silu, FLOAT16 high-precision and high-performance scenarios, BFLOAT16 scenarios, quantization scenarios, and fake-quantization scenarios with experts or without expert groups are supported, and N1=K2.
  • In non-quantization scenarios, do not input quantization or pseudo-quantization parameters. In quantization scenarios, do not input pseudo-quantization parameters. In pseudo-quantization scenarios, do not input quantization parameters.
  • Parameter types in quantization scenarios: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are divided into two cases according to the type of [object Object].
    • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64, INT64, and FLOAT32.
    • When the data type of [object Object] is BFLOAT16, [object Object] supports only the BFLOAT16 data type.
    • The data type of [object Object] must be the same as that of [object Object].
  • Parameter types in quantization scenarios where the per-channel mode of [object Object] is supported: The data type of [object Object] is INT8, that of [object Object] is INT8, that of [object Object] is INT32, that of [object Object] is FLOAT32, and that of [object Object] is FLOAT32. Other parameter types are categorized into two cases according to the type of [object Object].
    • When the data type of [object Object] is FLOAT16, [object Object] supports the following data types: UINT64 and INT64.
    • When the data type of [object Object] is BFLOAT16, [object Object] supports only the BFLOAT16 data type.
    • The data type of [object Object] must be the same as that of [object Object].
  • The pseudo-quantization scenario supports two parameter types:
    • The data type of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, that of [object Object] is FLOAT16, and that of [object Object] is FLOAT16. [object Object] supports data types INT8 and INT4.
    • The data type of [object Object] is BFLOAT16, that of [object Object] is BFLOAT16, that of [object Object] is FLOAT32, that of [object Object] is BFLOAT16, and that of [object Object] is BFLOAT16. [object Object] supports data types INT8 and INT4.
  • When the data type of [object Object] or [object Object] is INT4, the last dimension of the shape must be an even number.
  • In pseudo-quantization scenarios, under per-group mode, G (number of groups) in [object Object] and [object Object] must be exactly divided by K1, and G (number of groups) in [object Object] and [object Object] must be exactly divided by K2.
  • In the BFLOAT16 non-quantization scenario, [object Object] can only be set to [object Object]. In the FLOAT16 non-quantization scenario, [object Object] can be set to [object Object] or [object Object]. In quantization or fake-quantization scenarios, [object Object] can be set to [object Object] or [object Object], but the setting does not take effect.
  • If [object Object] is set to [object Object] and there are experts ([object Object] not being null), the value of [object Object] must meet the following requirements: If both [object Object] and [object Object] are valid array indexes in [object Object], and [object Object] is greater than [object Object], then the value of a jth element in [object Object] is greater than or equal to the value of an ith element in [object Object].

• [object Object]Atlas inference accelerator cards[object Object]:

  • Only the non-expert scenario is supported.
  • N1 must be equal to K2.

[object Object]

This fusion operator API supports only the aclnn single-operator calling mode, which is showcased in the following example. For details, see .