Description

Based on the given vocabulary probability, top-k, and top-p, the first k vocabulary probabilities are selected as candidates, and then vocabularies whose probabilities are higher than top-p are selected. Different postprocessing policies are used based on the value of TopkToppSamplingType, finally, the most appropriate word sequence number and the corresponding probability are selected as the output.

Top-k sampling: Select k tokens as candidates, and randomly select one from the most possible k options based on their likelihood scores.

Top-p sampling: Select a candidate from a minimum set of candidates whose cumulative probability is greater than or equal to the probability p from tokens.