Full/Incremental Separation

Description

When the model uses paged attention, select SelfAttention and set calcType to PA_ENCODER for the full phase, and select PagedAttention for the incremental phase. When the model uses the traditional flash attention, SelfAttention is used in both the incremental and full phases. In this case, to improve the computing efficiency, you can select different calcTypes in the full and incremental phases.

How to Enable

Full phase: Set calcType to ENCODER.

Incremental phase: Set calcType to DECODER.

Constraints

None