Mask Types

The mask types are not independent features. The mask of the paged attention operator is complex. For ease of understanding, the mask types are described separately.

maskType	Hardware Type	Dimension	Remarks
UNDEFINED	No mask	No mask	All-zero mask
MASK_TYPE_NORM	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[batch, 1, max_seq_len], [1, max_seq_len], or [max_seq_len, max_seq_len]	Inverted triangle mask
MASK_TYPE_NORM	Atlas inference products	[batch, max_seq_len / 16, 16, 16] or [1, max_seq_len / 16, 16, 16]	Inverted triangle mask
MASK_TYPE_ALIBI	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[batch, num_head, 1, max_seq_len] or [num_head, 1, max_seq_len]	alibi mask
MASK_TYPE_ALIBI	Atlas inference products	[batch * num_head, max_seq_len / 16, 16, 16] or [num_head, max_seq_len / 16, 16, 16]	alibi mask
MASK_TYPE_SPEC	Atlas A2 training products/Atlas A2 inference products Atlas A3 inference products/Atlas A3 training products	[num_tokens, max_seq_len]	Parallel decoding mask
MASK_TYPE_SPEC	Atlas inference products	[1, max_seq_len / 16, num_tokens, 16]	Parallel decoding mask
MASK_TYPE_MASK_FREE	Atlas inference products	[1, 8, 128, 16]	Fixed inverted triangle mask of 128 × 128.

For the Atlas inference products, max_seq_len should be 16-aligned, and the division in the dimension description is ceil div.

Parent topic: Functions