RopeQConcatOperation
Applicable Products
|
Hardware Model |
Supported or Not |
|---|---|
|
Atlas 350 accelerator card |
x |
|
|
x |
|
|
√ |
|
|
x |
|
|
x |
|
|
x |
Description
Performs the concat operation after rotary position embedding.
Formula
Rope calculation formula:
Computational Graphs
The following figure shows the kernel calculation process.
Definition
1 2 3 |
struct RopeQConcatParam { uint8_t rsv[16] = {0}; }; |
Parameters
|
Member |
Type |
Default Value |
Value Range |
Mandatory or Not |
Description |
|---|---|---|---|---|---|
|
rsv |
uint8_t[] |
{0} |
[0] |
No |
Reserved parameter. |
Input
|
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
|
q |
[ntokens, hidden_size_q] |
float16/bf16 |
ND |
2D. |
|
cos |
[ntokens, head_dim] |
Same as q |
ND |
2D. |
|
sin |
[ntokens, head_dim] |
Same as q |
ND |
2D. |
|
ConcatInput |
[ntokens, head_num, concat_size] |
Same as q |
ND |
3D. |
Output
|
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
|
ropeQConcat |
[ntokens, head_num, head_dim+concat_size] |
Same as q |
ND |
3D. |
Constraints
- hidden_size_q = head_dim × head_num.
- head_dim × sizeof(dtype) needs to be 32-byte aligned, that is, head_dim needs to be an integer multiple of 16 and less than or equal to 64.
- concat_size × sizeof(dtype) needs to be 32-byte aligned, that is, concat_size needs to be an integer multiple of 16.
- head_dim × 26 + concat_size × 2 < maxUbSize(196352) must be guaranteed, and there is no restriction on head_num.