AlltoallvcAllGatherGroupMatmul Input and Output

input

[m,k]

Floating point: float16/bf16

Quantization: int8

ND

Matrix A for matrix multiplication.

weight

[localExpertNums, k, n]

Floating point: float16/bf16

Quantization: int8

ND

Weight of matrix B for matrix multiplication.

bias

[n]

Quantization: int32

ND

Added bias matrix. Input when hasBias is true.

deqScale

[n*localExpertNums]

Quantization: int64/float32

ND

Scale of dequantization. This parameter is required only for quantization.

If the data type is float16 in the output, only int64 is supported. If the data type is bf16 in the output, both int64 and float32 are supported.

residual

[n]

float16/bf16

ND

Residual, which is added to the final output result.

dequantPerTokenScale

[m]

Quantization: float32

ND

Scale of perToken dequantization. This parameter is required only for perToken quantization.

globalTokensPerExpertMatrix

[ep, ep*localExpertNums]

int32

ND

Alltoallvc communication matrix.

maxOutputSize

[m']

int32

ND

Shape used to infer the output.

output

[m',n]

float16/bf16

ND

Output tensor.