[object Object][object Object][object Object]undefined[object Object]

• Updates the [object Object] and [object Object] at the specified position in the KvCache.

• Input and output support the following scenarios:

  • Scenario 1:

    [object Object]

  • Scenario 2:

    [object Object]

    The [object Object] and [object Object] can be different or the same.

  • Scenario 3:

    [object Object]

  • Scenario 4:

    [object Object]

  • Scenario 5:

    [object Object]
    • Scenario 6:
    [object Object]

  • The above scenarios are distinguished based on the constructed parameters. If the first type of parameter construction is met, it follows scenario 1; if the second type is met, it follows scenario 2; if the third type is met, it follows scenario 3; if the fourth type is met, it follows scenario 4; if the fifth type is met, it follows scenario 5; and if the sixth type is met, it follows scenario 6. Scenarios 1, 2, and 6 do not have the three optional parameters: [object Object], [object Object], and [object Object]. Scenario 4 does not have the optional parameter [object Object].

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] only support scenarios 1, 2, 4, 5, and 6.

[object Object]

Each operator has calls. First call [object Object] to obtain the required workspace size for computation and the executor that includes the operator's computation process. Then call [object Object] to perform the computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • [object Object] (aclTensor*, computation input): Device-side aclTensor, supports 3D or 4D. The key value to be updated and the key of multiple tokens in the current step. Data types supported include FLOAT16, FLOAT, BFLOAT16, INT8, UINT8, INT16, UINT16, INT32, UINT32, HIFLOAT8, FLOAT8_E5M2, and FLOAT8_E4M3FN, supports ND.
    • [object Object]Atlas A3 Training Series Products/Atlas A3 Inference Series Products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The data types supported are only FLOAT16, BFLOAT16, and INT8.
  • [object Object] (aclTensor*, computation input/output): Device-side aclTensor, only supports 4 dimensions, the key cache that needs to be updated, and the key cache of the current layer. The data type and format are consistent with the key.
  • [object Object] (aclTensor*, computation input): Device-side aclTensor, the storage offset of each token key or value in the cache. Supported data types include INT32, INT64, and the supports ND.
  • [object Object] (aclTensor*, computation input): Device-side aclTensor, supports 0D, 3D, or 4D. In non-0D cases, the shape is consistent with the key. The value to be updated, representing the value of multiple tokens in the current step, with data type and format consistent with the key.
  • [object Object] (aclTensor*, computation input/output): Device-side aclTensor, supports 0-dimensional or 4-dimensional. In non-0-dimensional cases, the shape is consistent with that of [object Object]. The value cache that needs to be updated and the value cache of the current layer. The data type and format are consistent with value.
  • [object Object] (aclTensor*, optional computation input): Device-side aclTensor, compression amount. Data type is consistent with [object Object], and supports ND.
  • [object Object] (aclTensor*, optional computation input): Device-side aclTensor, the compression starting point for each batch and each head. The data type is consistent with [object Object], and the supports ND.
  • [object Object] (aclTensor*, optional input): Device-side aclTensor, the actual seqLens for each batch. The data type is consistent with [object Object], and the supports ND.
  • [object Object] (char*, computation input): The [object Object] on the host side represents the memory layout format of [object Object] and [object Object].
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When passing a null pointer or [object Object], only the ND memory layout format is supported. When passing [object Object], only the [object Object] memory layout format is supported.
  • [object Object] (char*, computation input): The [object Object] on the host side represents the state of the updated key and value.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: When passing a null pointer or [object Object], it indicates that the updated key and value are in a non-compressed and continuous state. When passing [object Object], it indicates that the updated key and value are in a compressed state based on the Alibi structure. When passing [object Object], it indicates that the updated key and value are in a compressed state based on the Rope structure. When passing [object Object], it indicates that the updated key and value are in a compressed state based on the Omni structure. When passing [object Object], it indicates that the updated key and value are in a non-compressed but non-continuous state.
  • [object Object] (aclIntArray *, computation input): The strides of key and value in non-contiguous states, with an array length of 2. The values should be greater than 0.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Only effective when [object Object] is [object Object], representing [object Object] and [object Object] respectively.
  • [object Object] (aclIntArray *, computation input): The offsets of key and value in a non-contiguous state, with an array length of 2. The values should be greater than 0.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Only effective when [object Object] is [object Object], representing [object Object] and [object Object] respectively.
  • [object Object] (uint64_t*, computation output): Returns the size of the workspace that the user needs to allocate on the device side.
  • [object Object] (aclOpExecutor**, computation output): Returns the op executor, which includes the operator computation process.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Parameters:

  • [object Object] (void*, input): The memory address of the workspace allocated on the device side.
  • [object Object] (uint64_t, input): The size of the workspace allocated on the device side, obtained by the first-phase API of [object Object].
  • [object Object] (aclOpExecutor*, input): The op executor, which includes the operator computation process.
  • [object Object] (aclrtStream, input): Specifies the stream for task execution.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic Computation:
  • [object Object] defaults to deterministic implementation.
• In addition to key and value, input parameters do not support non-continuity;
• The data types of key, value, keyCacheRef, and valueCacheRef must be consistent;
• The data types of slotMapping, compressLensOptional, compressSeqOffsetOptional, and seqLensOptional must be consistent;
• The value range of slotMapping is [0, num_blocks * block_size - 1], and the element values within slotMapping are guaranteed to be unique; correctness is not guaranteed in case of duplication.
• When both key and value are 3-dimensional, the first two dimensions of their shapes must be the same;
• When both key and value are 4-dimensional, the first three dimensions of the key and value shapes must be the same, and the third dimension of keyCacheRef and valueCacheRef must be 1;
• When the key and value are 4-dimensional, [object Object] and [object Object] are required parameters; when the key and value are 3-dimensional, [object Object], [object Object], and [object Object] are optional parameters;
• When both key and value are 4-dimensional, [object Object] is 2-dimensional, and the first dimension of [object Object] equals the first dimension of key as batch, and the second dimension of [object Object] equals the third dimension of key as [object Object] (corresponding to scenario three);
• When both key and value are 4-dimensional, seqLensOptional is one-dimensional, and the value of seqLensOptional equals the first dimension of key as batch (corresponding to scenario three);
• When both key and value are 3-dimensional and seqLensOptional exists, the sum of all values in seqLensOptional equals the first dimension of key as num_blocks (corresponding to scenarios four and five);
• Each element value in [object Object] and [object Object] must satisfy the formula: [object Object] (corresponding to scenarios three, four, and five).

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]