PadWithHiddenStateOperation

Applicable Products

Product	Supported
Atlas 350 accelerator cards	x
Atlas A3 inference products/Atlas A3 training products	x
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

PadWithHiddenState is a pad with a hidden layer. This operator is used to fill a variable-length sequence with a fixed length to facilitate batch processing and model training.

Definition

struct PadWithHiddenStateParam {
    std::vector<int32_t> qSeqLen;
    int32_t  maxSeqLen = 4096;
    uint8_t rsv[12] = {0};
};

Parameters

Member	Type	Default Value	Description
qSeqLen	std::vector<int32_t>	-	Actual input length of each batch. The number of elements is batchSize. The value of batchSize cannot exceed 32.
maxSeqLen	int32_t	4096	Maximum input length in qSeqLen. The value cannot exceed 4096.
rsv[12]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[nTokens, hiddenSize]	float16	ND	Tensor without redundancy before pad. The shape is the same as that of the unpad output.

Formula:
$\text{[math]}$

This formula is used to calculate the total number of tokens in all sequences in a batch.
Example:
batchSize = 3

qSeqlen = [5, 10, 7]

nTokens = 5 +10 + 7 = 22

Output

Parameter	Dimension	Data Type	Format	Description
output	[batchSize, maxSeqLen, hiddenSize]	float16	ND	Redundant tensor after pad. The shape is the same as that of the unpad input.

Parent topic: atb/train_op_params.h