GroupedMatmulTransFusionPass
Description
Mode 1: Deletes Transpose or TransposeD from the graph and encodes the weight transformation information to operator attributes. The figure below shows the details.
This fusion mode supports the following products:
Atlas 350 Accelerator Card
Mode 2: Deletes Reshape, Transpose/TransposeD, and Reshape connected to the weight from the graph and encodes the weight transformation information to operator attributes. The figure below shows the details.
This fusion mode supports the following products:
Atlas 350 Accelerator Card
Mode 3: In fake-quantization scenarios of Atlas 350 Accelerator Cards, deletes Reshape+Transpose/TransposeD+Reshape connected to weight and antiquantScale from the graph and encodes the weight transformation information to operator attributes. The figure below shows the details.
Mode 4: In MX quantization scenarios of Atlas 350 Accelerator Card, deletes Transpose/TransposeD from the graph and encodes the x and weight transformation information to operator attributes. See the following figure.
In MX quantization scenarios of Atlas 350 Accelerator Card, scale must follow the same transformation as weight, while pertokenScale must follow the transformation applied to x.
Restrictions
- Only the scenario where x, weight, and y are single tensors is supported (single tensor indicates that there is only one tensor in the tensorList input).
- Transpose/TransposeD supports transpose on the second and third axes.