Approximate Exp Computing
Description
The exp uses m8v2 approximate computing to improve performance.
During attention calculation, the exp operation takes a long time for a single operator. Therefore, fast exp can be used for fast calculation.
Based on this theory, the exp m8v2 solution is proposed to improve precision while maintaining performance:
The polynomial can use the Qin Jiushao algorithm to reduce the number of operations:
The coefficients are determined after numerical optimization.
How to Enable
Set the kernelType parameter to KERNELTYPE_EXP_M8V2.
- Operator input list
Parameter
Dimension
Data Type
Format
cpu/npu
Description
Q
[ntokens, heads, headSize]
float16
ND
npu
nTokens: sum(qSeqlen), and the sum is rounded up to the nearest multiple of 16.
K
[ntokens, kvHead, headSize]
float16
ND
npu
-
V
[ntokens, kvHead, headSize]
float16
ND
npu
The v shape is the same as that of k.
Attention_mask
Compressed mask: maxSeqlen = 128[1, 128/16, 128, 16]
float16
NZ
npu
This parameter is not passed when maskType is MASK_TYPE_UNDEFINED (0).
seqlen
[batch]
int32/uint32
ND
npu
-
- Operator output list
Parameter
Dimension
Data Type
Format
cpu/npu
Description
output
[nTokens, head_num, head_size]
float16
ND
npu
Output.
Constraints
- headSize must be equal to headSizeV. MLA is not supported.
- headSize must be 16-pixel aligned.
- m8v2 is supported only on the
Atlas inference products . - Only configuring masktype to MASK_TYPE_UNDEFINED or MASK_TYPE_NORM is supported.
- Only configuring calcType to PA_ENCODER is supported.
- headNum != kvHeadNum is supported.
- qkScale normalization is supported.
- qScale scaling is not supported.
- quantType, outDataType, batchRunStatusEnable, isTriuMask, clampType, clampMin, clampMax, kvcacheCfg, scaleType, mlaVHeadSize, cacheType and windowSize support only the default values.