Computable Batch Control

Description

Specifies batches for attention computation.

How to Enable

Set batchRunStatusEnable to true and pass batchRunStatus as the input tensor.

Tensor whose batchRunStatus is 0 or 1. The value 0 indicates that the batch is not involved in the computation, and the value 1 indicates that the batch is involved in the computation.

Constraints

• Not supported by the Atlas inference products.
• When this function is enabled, the dimensions of cacheK and cacheV are [batch, maxSeqLen, hiddenSize].
• PA_ENCODER is not supported.
• PREFIX_ENCODER is not supported.