Input and Output When calcType Is CALC_TYPE_PREFILL and maskType Is MASK_TYPE_CAUSAL_MASK
Parameter |
Dimension |
Data Type |
Format |
cpu/npu |
Description |
|---|---|---|---|---|---|
qNope |
[tokenNum, headNum * embeddimV]/[tokenNum, headNum, embeddimV] |
float16/bf16 |
ND |
NPU |
Input. Query without positional encoding. |
qRope |
[tokenNum, headNum * 64] / [tokenNum, headNum, 64] |
float16/bf16 |
ND |
NPU |
Input. Query with rotational positional encoding. |
kNope |
[batch, maxSeqLen, kvHeadNum * embeddimV] |
float16/bf16 |
ND |
NPU |
Input. Key without positional encoding. |
kRope |
[batch, maxSeqLen, kvHeadNum * 64] |
float16/bf16 |
ND |
NPU |
Input. Key with rotational positional encoding. |
vNope |
[batch, maxSeqLen, kvHeadNum * embeddimV] |
float16/bf16 |
ND |
NPU |
Input. Value without positional encoding. |
qSeqLen |
[batch] |
int32 |
ND |
CPU |
Input. Query length. |
kvSeqLen |
[batch] |
int32 |
ND |
CPU |
Input. Key length. |
attenOut |
[tokenNum, headNum * embeddimV]/[tokenNum, headNum, embeddimV] |
float16/bf16 |
ND |
NPU |
Attention output. |