Parameters

topkToppSamplingType

TopkToppSamplingType

SINGLE_TOPK_SAMPLING

Sampling type.

• SAMPLING_UNDEFINED: not defined and not used as the sampling type.
• SINGLE_TOPK_SAMPLING: non-batch random seed, top-k. BATCH_TOPK_MULTINOMIAL_SAMPLING: batch random seed, top-k multinomial sampling.
• BATCH_TOPK_EXPONENTIAL_SAMPLING: batch random seed, top-k exponential sampling.
• BATCH_TOPK_MULTINOMIAL_LOGPROBS_SAMPLING: batch random seed, top-k multinomial sampling, and logProbs added to the output. It is supported only by the Atlas A2 training products/Atlas A2 inference products.
• BATCH_TOPK_EXPONENTIAL_LOGPROBS_SAMPLING: batch random seed, top-k exponential sampling, and logProbs added to the output. It is supported only by the Atlas A2 training products/Atlas A2 inference products.
• SAMPLING_MAX: maximum value, which is not used as the sampling type.

randSeed

uint32_t

0

Random seed used for random sampling in top-p . The default value is 0. Used when topktoppSamplingtype = SINGLE_TOPK_SAMPLING.

topk

uint32_t

100

Number of words reserved in top-k. The value must be less than the number of words in the vocabulary. The default value is 100. The value of topk must be greater than 0 and less than or equal to the size of the last dimension of the input probs. Used when topktoppSamplingtype = SINGLE_TOPK_SAMPLING.

randSeeds

std::vector<uint32_t>

-

Random number seed used for random sampling in the top-p phase of each batch. The dimension is the same as the batch size. Used when topktoppSamplingtype = BATCH_TOPK_MULTINOMIAL_SAMPLING.

logProbsSize

int32_t

0

It is an int-type variable of the scalar. The value range is (0, 16384]. This parameter is valid only when the enumerated value is BATCH_TOPK_EXPONENTIAL_LOGPROBS_SAMPLING or BATCH_TOPK_MULTINOMIAL_LOGPROBS_SAMPLING.

rsv[12]

uint8_t

{0}

Reserved