LogN Scaling

For the Qwen long sequence scenario, LogN is used to scale attention.

Formula:

$\text{[math]}$

m is a training length, and n is a prediction length.

Set scaleType to infer::PagedAttentionParam::SCALE_TYPE_LOGN.

When LogN scaling is used, add an input tensor named logN to PagedAttentionOperation. The specifications are as follows.

Parameter	Dimension	Data Type	Format	cpu or npu
logN	Incremental phase: [batch size]	Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products: float32 Atlas inference products: float16	ND	NPU

Parameter

Dimension

Data Type

Format

cpu or npu

logN

Incremental phase: [batch size]

Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products: float32

Atlas inference products: float16

NPU

Other input tensors remain unchanged.

None

Parent topic: Functions