KvCacheExtParam

Represents the extended parameters passed when the KV cache pull or push APIs are called.

struct KvCacheExtParam {
  std::pair<int32_t, int32_t> src_layer_range = {-1, -1};  // Layer range at the source side during KV transmission.
  std::pair<int32_t, int32_t> dst_layer_range{-1, -1};  // Layer range at the destination side during KV transmission
  uint8_t tensor_num_per_layer = 2U;                       // Number of tensors per layer during KV transmission.
  uint8_t reserved[127];                                   // Reserved.
}

Parent topic: Data Structures