Input in KV TensorList Format

Description

Input cacheK and cacheV as tensorlist, that is, the level-2 pointer structure.

How to Enable

How to construct the variantPack in this scenario:

Definition of the atb tensor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

struct Tensor {
    //! \brief Tensor description
    TensorDesc desc;
    //! \brief Tensor NPU memory address
    void *deviceData = nullptr;
    //! \brief Tensor CPU memory address
    void *hostData = nullptr;
    //! \brief "deviceData" or "hostData" points to the memory size of the content.
    uint64_t dataSize = 0;
};

For the input tensors in variantPack, when the tensor list needs to be enabled, set deviceData of the corresponding tensor to nullptr and point hostData to the constructed tensorList.

Constraints

This function takes effect only when the kv-bypass function is enabled (kvcacheCfg is set to K_BYPASS_V_BYPASS) and calcType is set to DECODER.