[object Object]

[object Object][object Object]undefined[object Object]

• Operator function: updates the key at the specified position in the KCache.

• Formulas:

  • Scenario 1:

    [object Object]
    $$keyCache = slotMapping(key)$$

  • Scenario 2:

    [object Object]
    $$\begin{aligned} keyCache =\ & slotMapping(key[: compressSeqOffset], \\ & ReduceMean(key[compressSeqOffset : compressSeqOffset + compressLens]), \\ & key[compressSeqOffset + compressLens : seqLens]) \end{aligned}$$

  • Scenario 3:

    [object Object]
    $$keyCache = slotMapping(key[seqLens - compressLens : seqLens])$$

  The preceding scenarios are distinguished based on the constructed parameters. If the first input parameter is constructed, scenario 1 is used. If the second input parameter is constructed, scenario 2 is used. If the third input parameter is constructed, scenario 3 is used. In scenario 1, the compressLensOptional, seqLensOptional and compressSeqOffsetOptional parameters are unavailable. In scenario 3, the compressSeqOffsetOptional parameter is unavailable.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Return Value

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return Value

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing: The default deterministic implementation of aclnnScatterPaCache is used.
• The variables used by shape in the parameter description are described as follows:
  • batch: number of input sequences (number of samples processed at a time). The value is a positive integer.
  • seq_len: length of the sequence. The value is a positive integer.
  • num_head: number of heads in multi-head attention. The value is a positive integer.
  • k_head_size: feature dimension of the key in each attention head (length of the key in a single head). The value is a positive integer.
  • num_blocks: total number of blocks pre-allocated in keyCache, which is used to store the key data of all sequences. The value is a positive integer.
  • block_size: number of tokens contained in each cache block. The value is a positive integer.
• Input value range restriction: Each element value in seqLensOptional and compressLensOptional must meet the following formula: reduceSum(seqLensOptional[i] - compressLensOptional[i] + 1) <= num_blocks * block_size (corresponding to scenarios 2 and 3).

[object Object]

The following example is for reference only. For details, see .

• Ascend 950PR/Ascend 950DT:

  [object Object]