[object Object][object Object][object Object]undefined[object Object]

• Description: Performs top-k and top-p sampling and filtering on the original input logits.

• Formula:

  • Sorts the input logits in ascending order along the last axis to obtain the corresponding sorting results sortedValue and sortedIndices. $$sortedValue, sortedIndices = sort(logits, dim=-1, descending=false, stable=true)$$
  • Calculates the reserved threshold (the kth largest value). $$topKValue[b][v] = sortedValue[b][sortedValue.size(1) - k[b]]$$
  • Generates the mask to be filtered for top-k. $$topKMask = sortedValue < topKValue$$
  • Sets the part that is less than the threshold to -Inf using topKMask.
  $$sortedValue[b][v] = \begin{cases} -Inf & \text{topKMask[b][v]=true}\\ sortedValue[b][v] & \text{topKMask[b][v]=false} \end{cases}$$
  • Converts the data filtered by top-k into probability distribution along the last axis using softmax. $$probsValue = softmax(sortedValue, dim=-1)$$
  • Calculates the cumulative probability along the last axis (starting cumulation from the smallest probability). $$probsSum = cumsum(probsValue, dim=-1)$$
  • Generates the top-p mask. The positions whose cumulative probability is less than or equal to (1 – p) need to be filtered out, and at least one element must be reserved in each batch. $$topPMask[b][v] = probsSum[b][v] <= 1-p[b]$$ $$topPMask[b][-1] = false$$
  • Sets the part that is less than the threshold to -Inf using topPMask.
  $$sortedValue[b][v] = \begin{cases} -Inf & \text{topPMask[b][v]=true}\\ sortedValue[b][v] & \text{topPMask[b][v]=false} \end{cases}$$
  • Restores the filtered result to the original order based on sortedIndices. $$out[b][v] = sortedValue[b][sortedIndices[b][v]]$$ Where, $0 \le b \lt logits.size(0), 0 \le v \lt logits.size(1)$.

[object Object]

Each operator has calls. First, aclnnApplyTopKTopPGetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnApplyTopKTopP is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic compute:
  • aclnnApplyTopKTopP defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .