LayerSynchronizer

An abstract class that users need to inherit from and implement the related APIs. Currently, this class contains only the synchronize_layer API.

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Waits until the execution of the specified layer of the model is complete. You need to inherit LayerSynchronizer and implement this API.

This API is called during the execution of transfer_cache_async in KvCacheManager. When the API returns a success status, the cache transmission for the current layer begins.

Prototype

synchronize_layer(layer_index: int, timeout_in_millis: Optional[int]) -> bool

Parameters

Parameter	Data Type	Description
layer_index	int	Index of a layer.
timeout_in_millis	Optional[int]	Timeout interval, which is not supported currently.

Example

This API is not directly called by users. Instead, it is called by KvCacheManager as a callback function.

Returns

In normal cases, a message is returned, indicating that whether the synchronization is successful.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Constraints

None

Parent topic: API Reference