Input in KV TensorList Format

Description

Inputs a tensor list, that is, the level-2 pointer structure, rather than the entire tensor for cacheK and cacheV.

How to Enable

Constructing variantPack:

Defining the ATB Tensor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

struct Tensor {
    //! \brief Tensor description
    TensorDesc desc;
    //! \brief Tensor NPU memory address
    void *deviceData = nullptr;
    //! \brief Tensor CPU memory address
    void *hostData = nullptr;
    //! \brief "deviceData" or "hostData" points to the memory size of the content.
    uint64_t dataSize = 0;
};

For the input tensors in variantPack, when the tensor list needs to be enabled, set deviceData of the corresponding tensor to nullptr and point hostData to the constructed tensorList.

Constraints

This function takes effect only when kv-bypass is enabled (kvcacheCfg is set to K_BYPASS_V_BYPASS) and calcType is set to DECODER.