per_device_tensor_addrs

Applicable Products

Note: For the Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Obtains the KV cache address.

Prototype

1
2

@property
per_device_tensor_addrs() -> List[List[int]]

Parameters

None

Example

1
2
3
4

...
kv_cache = kv_cache_manager.allocate_cache(cache_desc, cache_keys)
# The two-layer list structure uses the single-process multi-device design. In single-process single-device mode, the address of the first device is obtained.
print(kv_cache.per_device_tensor_addrs[0])

Returns

In normal cases, the address of the KVCache type is returned.

Constraints

None