Mask Types

The mask types are not independent features. The mask of the paged attention operator is complex. For ease of understanding, the mask types are described separately.

maskType	Hardware Type	Dimension	Remarks
UNDEFINED	No mask	No mask	All-zero mask.
MASK_TYPE_NORM	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[batch, 1, max_seq_len], [1, max_seq_len], or [max_seq_len, max_seq_len]	Inverted triangle mask.
MASK_TYPE_NORM	Atlas inference products	[batch, max_seq_len / 16, 16, 16] or [1, max_seq_len / 16, 16, 16]	Inverted triangle mask.
MASK_TYPE_ALIBI	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[batch, num_head, 1, max_seq_len] or [num_head, 1, max_seq_len]	alibi mask.
MASK_TYPE_ALIBI	Atlas inference products	[batch * num_head, max_seq_len / 16, 16, 16] or [num_head, max_seq_len / 16, 16, 16]	alibi mask.
MASK_TYPE_SPEC	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[num_tokens, max_seq_len]	Parallel decoding mask.
MASK_TYPE_SPEC	Atlas inference products	[1, max_seq_len / 16, num_tokens, 16]	Parallel decoding mask.
MASK_TYPE_MASK_FREE	Atlas inference products	[1, 8, 128, 16]	Fixed inverted triangle mask of 128 × 128.
MASK_TYPE_NORM_COMPRESS	Atlas inference products	ND: [2048,2048]	Inverted triangular compressed mask. The mask data type can only be float16. The upper triangle (excluding the diagonal) is a large negative value (invisible, -60000 or -10000 is recommended), and the lower triangle and diagonal are 0 (visible). All batches/heads share the same 2048 x 2048 mask. Input in ND format.

For Atlas inference products, max_seq_len should be a multiple of 16, and the division in the dimension description is ceil div.

Parent topic: Functions