High Precision

Description

During the attention computation, the Q∙K^T result may overflow the float16 data type. As a result, the NAN value is displayed in the output tensor of the operator. In this case, you can enable this function so that the operator uses the float32 data type to carry the intermediate result.

How to Enable

Set kernelType to KERNELTYPE_HIGH_PRECISION.

Constraints

When enabling this function, change -inf in the input mask to 1.

High precision setting takes effect only on Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.