Constraints
- ntokens = sum(seqlen[i]), i=0, 1...batch-1.
- If the last dimensions of the key and value are different, only
Atlas A2 training products /Atlas A2 inference products andAtlas A3 inference products /Atlas A3 training products are supported. In this scenario, keyCache, valueCache, keyCacheOut, and valueCacheOut support only the ND format. - The data types of key, value, keyCache, valueCache, keyCacheOut, and valueCacheOut must be the same.
- The data formats of keyCache, valueCache, keyCacheOut, and valueCacheOut must be the same.
- The single-input single-output function is supported only by the
Atlas A2 training products /Atlas A2 inference products andAtlas A3 inference products /Atlas A3 training products and does not support compression. - In the compression scenario, the cache size must be greater than the size after compression.
- Currently, only the last two dimensions of key and value can be continuous.
Parent topic: ReshapeAndCacheWithStrideOperation