[object Object]

[object Object][object Object]undefined[object Object]

• API function: Updates the [object Object] and [object Object] at a specified position in KvCache.

• The input and output support the following scenarios:

  • Scenario 1:

    [object Object]

  • Scenario 2:

    [object Object]

    [object Object] and [object Object] can be the same or different.

  • Scenario 3:

    [object Object]

  • Scenario 4:

    [object Object]

  • Scenario 5:

    [object Object]
    • Scenario 6
    [object Object]

• The preceding scenarios are distinguished based on the constructed parameters. If the first type of input parameter is constructed, scenario 1 is used. If the second type of input parameter is constructed, scenario 2 is used. If the third type of input parameter is constructed, scenario 3 is used. If the fourth type of input parameter is constructed, scenario 4 is used. If the fifth type of input parameter is constructed, scenario 5 is used. If the sixth type of input parameter is constructed, scenario 6 is used. In scenarios 1, 2, and 6, the optional parameters compressLensOptional, seqLensOptional, and compressSeqOffsetOptional are not available. In scenario 4, the optional parameter compressSeqOffsetOptional is not available.

• For [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Currently, the [object Object] and [object Object] modes are supported.

[object Object]

Each operator has calls. First, aclnnScatterPaKvCacheGetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnScatterPaKvCache is called to perform computation.

[object Object]

• Parameter Description

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The input key, keyCacheRef, value, and valueCacheRef do not support the FLOAT, UINT8, INT16, UINT16, INT32, UINT32, HIFLOAT8, FLOAT8_E5M2 and FLOAT8_E4M3FN data types.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameter Description

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
  • The data types of key, value, keyCacheRef, and valueCacheRef must be the same.
  • The data types of slotMapping, compressLensOptional, compressSeqOffsetOptional, and seqLensOptional must be the same.
  • The value range of slotMapping is [0, num_blocks*block_size-1]. The values of elements in slotMapping must be unique. If the values are duplicate, the correctness cannot be ensured.
  • If both key and value are 3-dimensional, the first two dimensions of key and value must have the same shape.
  • If both key and value are 4-dimensional, the first three dimensions of key and value must have the same shape, and the third dimension of keyCacheRef and valueCacheRef must be 1.
  • If both key and value are 4-dimensional, compressLensOptional and seqLensOptional are mandatory. If both key and value are 3-dimensional, compressLensOptional, compressSeqOffsetOptional, and seqLensOptional are optional.
  • If both key and value are 4-dimensional, slotMapping is 2-dimensional, the value of the first dimension of slotMapping is equal to the value of the first dimension of key (batch), and the value of the second dimension of slotMapping is equal to the value of the third dimension of key (num_head) (corresponding to scenario 3).
  • If both key and value are 4-dimensional, seqLensOptional is 1-dimensional, and the value of seqLensOptional is equal to the value of the first dimension of key (batch) (corresponding to scenario 3).
  • If both key and value are 3-dimensional and seqLensOptional is available, the sum of all values in seqLensOptional is equal to the value of the first dimension of key (num_blocks) (corresponding to scenarios 4 and 5).
  • The value of each element in seqLensOptional and compressLensOptional must meet the following formula: reduceSum(seqLensOptional[i] - compressLensOptional[i]) <= num_blocks * block_size (corresponding to scenarios 3, 4, and 5).
  • When cacheModeOptional is set to PA_NZ, the second-to-last dimension of keyCacheRef and valueCacheRef must be less than UINT16_MAX (corresponding to scenario 1).

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]