Non-Batch Random Seed and Top-K Sampling

A single randSeed and top-k are used. All batches of the input vocabulary probabilities are truncated by using a unified top-k value, and then top-p sampling is performed on all batches of the truncated vocabulary probabilities by using a unified randSeed.

Flowchart

Figure 1 Non-batch random seed top-k sampling process