High Precision

Description

During the attention computation, the Q∙K^T result may overflow the float16 data type. As a result, the NAN value is displayed in the output tensor of the operator. In this case, you can enable this function so that the operator uses the float32 data type to carry the intermediate result.

How to Enable

Set kernelType to KERNELTYPE_HIGH_PRECISION.

Restrictions

When this function is enabled, -inf in the input mask needs to be changed to 1.

The high-precision function takes effect only on the Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.