[object Object]

[object Object][object Object]undefined[object Object]

• Description:

  vLLM is a high-performance LLM inference and service framework that focuses on optimizing the inference efficiency of large-scale language models. Its core features include PageAttention and efficient memory management. The main function of the advance_step operator is to advance the inference step, that is, update the model status and generate new inputTokens, inputPositions, seqLens, and slotMapping in each generation step, improving the inference efficiency of vLLM.

• Formula:

  $$blockIdx is the index of the core where the current code is executed.$$ $$blockTablesStride = blockTables.stride(0)$$ $$inputTokens[blockIdx] = sampledTokenIds[blockIdx]$$ $$inputPositions[blockIdx] = seqLens[blockIdx]$$ $$seqLens[blockIdx] = seqLens[blockIdx] + 1$$ $$slotMapping[blockIdx] = (blockTables[blockIdx] + blockTablesStride * blockIdx) * blockSize + (seqLens[blockIdx] \% blockSize)$$

[object Object]

Each operator has calls. First, aclnnAdvanceStepV2GetWorkspaceSize is called to obtain the input parameters and compute the required workspace size based on the process. Then, aclnnAdvanceStepV2 is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

[object Object]: status code. For details, see .

The first-phase API implements input parameter verification. The following errors may be thrown:

[object Object][object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic compute:
  • aclnnAdvanceStepV2 defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .