FaUpdateOperation

Applicable Products

Hardware Model	Supported or Not
Atlas 350 accelerator card	x
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Updates the intermediate results lse and attention out output by the PA operators in each SP domain to global results.

Definition

struct FaUpdateParam {
    enum FaUpdateType {
        DECODE_UPDATE = 0, 
    };
    FaUpdateType faUpdateType = DECODE_UPDATE;
    uint32_t sp = 1;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Value Range	Mandatory or Not	Description
faUpdateType	FaUpdateType	DECODE_UPDATE	DECODE_UPDATE	No	Type of operations to be performed for a specific index. Currently, only the default value is supported.
sp	uint32_t	1	[1, 16]	No	Sequence parallelism.
rsv[64]	uint8_t	{0}	[0]	No	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
lse	[sp, batch * seqLen * headNum]	float	ND	lse calculated by each SP domain.
localout	[sp, batch * seqLen * headNum, head_size]	float	ND	Output calculated by each SP domain.

Output

Parameter	Dimension	Data Type	Format	Description
output	[batch * seqLen * headNum, head_size]	float	ND	Global output.

Constraints

The value range of head_size is [8, 512], and the value must be a multiple of 8.

Parent topic: atb/infer_op_params.h