MatMulBiasAddFusionPass

Description

Fuses the MatMul/MatMulV2/MatMulV3/BatchMatmul/BatchMatMulV2/BatchMatMulV3 and biasadd/add operators into the MatMul/MatMulV2/MatMulV3/BatchMatmul/BatchMatMulV2/BatchMatMulV3 operator.

Before: After:

Restrictions

• One of the two inputs of the add operator must have a dimension of 1.
• The value of bias must be the same as that of the last dimension of the MatMul, MatMulv2 or MatMulV3/BatchMatmul/BatchMatMulV2/BatchMatMulV3 output.
• The output dimension of MatMul/MatMulV2/MatMulV3 is 2.
• broadcast is not supported. (broadcast enables two tensors with different shapes to automatically expand to the same shape for element-wise operations. broadcast checks the shapes of the two tensors from right to left. If the dimension of one side is 1, it automatically expands to be the same as that of the other side.)
• If BiasAdd/Add is computed in FP16 before fusion, bias is accumulated inside matmul in FP32 after fusion. This leads to a precision improvement. If you disable MatMulBiasAddFusionPass, the precision remains unchanged.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card