Parallel Decoding

Description

In a conventional inference process, serial decoding needs to be performed on tokens one by one. As a result, time consumption is in direct proportion to a quantity of generated tokens. This disadvantage is especially obvious when step-by-step decoding is implemented. To solve this, lookahead is introduced. In the decode phase, multiple candidate tokens are obtained from the n-gram for parallel decoding, accelerating the model inference.

How to Enable

Pass qSeqLens as the input tensor, and set calcType to CALC_TYPE_SPEC.

• Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products: Set maskType to MASK_TYPE_NORM or MASK_TYPE_SPEC.
• Atlas inference products: Set maskType to UNDEFINED or MASK_TYPE_SPEC.

Constraints

None