[object Object][object Object][object Object]undefined[object Object]

• Description: Scans data in FFN when Attention and FFN are separately deployed. This operator receives data from the AttentionToFFN operator for data scanning and sorting.

  It is recommended that this operator be used with AttentionToFFN and FFNWorkerBatching.

  1. Receives data from the AttentionToFFN operator. The data is stored in the memory layout of the ScheduleContext structure. For details, see . This structure contains CommonArea, ControlArea, AttentionArea, and FfnArea. This API involves the CommonArea (used to store configuration information, such as session_num, micro_batch_num, micro_batch_size, and selected_expert_num), ControlArea (used to control whether the upper-layer process exits), and FfnArea (used to manage the input and output buffers required for the computation of this operator. The token_info_buf field is used to store the input information of this operator).

  2. Scans the information stored in [object Object]. When the communication data is ready, this operator starts data sorting. As shown in the following figure, the information of [object Object], [object Object], [object Object], and [object Object] are written to the [object Object], [object Object], [object Object], and [object Object], respectively, in the device memory.

  [object Object]
  3. Allows data to be used by the FFNWorkerBatching operator after the data is sorted.

• Formula:

  1. Perform initialization. The number of groups is computed based on the values of [object Object] and [object Object] of the input parameter [object Object].
  2. If the number of groups is 1, all data is processed synchronously. After all session data is ready, the data is sorted.
  3. If the number of groups is not 1, data is not processed synchronously. After the session data in the groups is ready, the data is sorted.

$$\text{Initialize:} \quad\text{group\_num} = \frac{\text{session\_num}}{\text{sync\_group\_size}}$$

$$ \text{Process} = \begin{cases} \text{check_all_session_ready()} \quad \text{data_reorganization()} & \text{if } \text{group_num} = 1 \ \text{check_all_sessions_of_group_ready()} \quad \text{data_reorganization()} & \text{otherwise} \end{cases} $$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• By default, [object Object] defaults to deterministic implementation . The non-deterministic implementation is not supported, and the deterministic computation configuration does not take effect.

[object Object]

The following example is for reference only. For details, see .