Batch Random Seed and topk logprobs Multinomial Sampling

Each batch of the vocabulary probabilities is truncated by using a different topk, and then multinomial sampling of the top-p is performed on each batch of the truncated vocabulary probabilities by using a different random seed.

Flowchart

Figure 1 topk logprobs multinomial sampling of the batch random seed