PadOperation
Description
Extracts the embedding vector of the last valid token in each batch for input_ids.
Application Scenarios
After padding (with zeros at the end), input_ids has a batch of token ID sequences. During output, the word vector of the last token is obtained from each batch and combined.
Example:
tmp_out [token_num,hidden_dim] is [37,16].
padding_offset[1, token_num] is [37].
seqlen[batch,1] is [[8],[13],[16]].
input_ids[3,16] is as follows:
[[1,2,3,4,5,6,7,8,0,0,0,0,0,0,0,0],[1,2,3,4,5,6,7,8,9,10,11,12,13,0,0,0],[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]]
max_seq_len=16
output[3,16] is as follows:
[[1,2,3,4,5,6,7,8,0,0,0,0,0,0,0,0],[1,2,3,4,5,6,7,8,9,10,11,12,13,0,0,0],[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]]
Application scenarios:
In natural language processing (NLP), the length of an input sentence is usually different. To process them in a batch, we need to pad them to the same length. A padding method is usually adding zeros (0) at the end of a sentence until all sentences have a same length.
Definition
struct PadParam {
uint8_t rsv[8] = {0};
};
Parameters
Member |
Type |
Default Value |
Description |
|---|---|---|---|
rsv[8] |
uint8_t |
{0} |
Reserved |
Input
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
tmp_out |
[token_num, hidden_dim] |
float16 |
ND |
Embedding vector corresponding to each token. |
padding_offset |
[1, token_num] |
int32 |
ND |
Same as the output padding_offset in UnpadOperation. |
seq_len |
[batch, 1] |
int32 |
ND |
Number of valid tokens in each batch. |
input_ids |
[batch, max_seq_len] |
int64 |
ND |
Token ID sequences (number determined by batch) after padding (after 0 is padded at the end). |
Output
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
output |
[batch, hidden_dim] |
float16 |
ND |
Embedding vector of the last valid token in each batch. |