UnpadWithHiddenStateOperation

Description

Restores a padded sequence to the original variable-length sequence.

Definition

struct UnpadWithHiddenStateParam {
    std::vector<int32_t> qSeqLen;
    int32_t maxSeqLen = 4096;
    uint8_t rsv[12] = {0};
};

Parameters

Member	Type	Default Value	Description
qSeqLen	std::vector<int32_t>	-	Actual SeqLen value of each batch. The number of elements is batchSize. The value of batchSize cannot exceed 32.
maxSeqLen	int32_t	-	Maximum SeqLen value. The value cannot exceed 4096.
rsv[12]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[batchSize, maxSeqLen, hiddenSize]	float16	ND	Redundant tensor before unpad. Its shape is the same as the pad's output shape.

Output

Parameter	Dimension	Data Type	Format	Description
output	[nTokens, hiddenSize]	float16	ND	Tensor without redundancy after unpad. Its shape is the same as the pad's input shape. nTokens is the sum of elements in the qSeqLen array.

Restrictions

Currently, only the Atlas A2 inference products is supported.

API Calling Example

qSeqlen ={2,3,1} // Actual input length of each batch

maxSeqlen=4 // Maximum input length in qSeqlen

The shape of x is (3, 4, 5). // batchSize is 3, the maximum input length is 4, and the size of the hidden layer is 5.

x=[
[1,2,3,4,5
6,7,8,9,10
0,0,0,0,0
0,0,0,0,0],
[11,12,13,14,15
16,17,18,19,20
21,22,23,24,25
0,0,0,0,0],
[26,27,28,29,30
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0]]

The shape of the output is (6,5). // The total number of tokens is 6, and the size of the hidden layer is 5.

x=[1, 2, 3, 4, 5
6, 7, 8, 9, 10
11, 12, 13, 14, 15
16, 17, 18, 19, 20
21, 22, 23, 24, 25
26, 27, 28, 29, 30]

Parent topic: atb/train_op_params.h