Precision Tuning

ge.customizeDtypes

Customizes the operator precision during model build. Other operators in the model are built according to ge.exec.precision_mode or ge.exec.precision_mode_v2. This parameter should be set to a path (including name of the configuration file), for example, /home/test/customize_dtypes.cfg.

Restrictions:

List the names or types of operators whose computing precision needs customization in the configuration file. Each operator occupies a line, and the operator type must be defined based on IR.
If both of the operator name and type are configured for an operator, the operator name applies during build.
The computing precision of an operator specified by this parameter does not take effect if the operator is fused during operator build.

The structure of the configuration file is as follows:

# Configuration by operator name
Opname1::InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
Opname2::InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
# Configuration by operator type
OpType::TypeName1:InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
OpType::TypeName2:InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...

The following is an example of the configuration file:

# Configuration by operator name
resnet_v1_50/block1/unit_3/bottleneck_v1/Relu::InputDtype:float16,int8,OutputDtype:float16,int8
# Configuration by operator type
OpType::Relu:InputDtype:float16,int8,OutputDtype:float16,int8

You can view the computing precision supported by an operator in the operator information library, which is stored in ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json by default.
Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann. xxx based on the actual product.
The data type specified by this option takes high priority, which may cause precision or performance degradation. If the specified data type is not supported, the build will fail.

Mandatory/Optional: optional

Effective level: session

ge.exec.modify_mixlist

Specifies the path and file name of the blocklist, trustlist, and graylist, the operators that allow precision degradation, and the operators that do not allow precision degradation when mixed precision is enabled. Set this parameter to the path including the file name. The file is in JSON format.

For the blocklist, trustlist, and graylist, you can view the value of the flag parameter under precision_reduce in the built-in tuning policy file of ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json. (Select Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.xxx based on the actual product.)

true (trustlist): Precision reduction is allowed in mixed precision mode.
false (blocklist): Precision reduction is not allowed in mixed precision mode.
Not specified (graylist): Operators on the graylist follow the same precision processing as its upstream operator.

Configuration example:

{"ge.exec.modify_mixlist", "/home/test/ops_info.json"};

You can specify the operator types in ops_info.json as shown below. Separate operators with commas (,).

{
  "black-list": {                  // Blocklist
     "to-remove": [                // Move an operator from the blocklist to the graylist.
     "Xlog1py"
     ],
     "to-add": [                   // Move an operator from the trustlist or graylist to the blocklist.
     "Matmul",
     "Cast"
     ]
  },
  "white-list": {                  // Trustlist
     "to-remove": [                // Move an operator from the trustlist to the graylist.
     "Conv2D"
     ],
     "to-add": [                   // Move an operator from the blocklist or graylist to the trustlist.
     "Bias"
     ]
  }
}

The operators in the preceding example configuration file are for reference only. The configuration should be based on the actual hardware environment and the built-in tuning policies of the operators. The following is an example of querying the blocklist, trustlist, and graylist:

"Conv2D":{
    "precision_reduce":{
        "flag":"true"
     }
},

true: trustlist; false: blocklist; not configured: graylist

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.precision_mode

Sets the operator precision mode. The value must be of the string type. This parameter cannot be used together with ge.exec.precision_mode_v2. You are advised to use ge.exec.precision_mode_v2.

Parameter values:

force_fp32/cube_fp16in_fp32out:
force_fp32 and cube_fp16in_fp32out have the same effect. This option indicates that the system selects different processing modes based on the operator type when the operator in the AI Core supports both the float32 and float16 data types. cube_fp16in_fp32out is newly added to the new version. For cube operators, this option has clearer semantics.
- For cube operators, the system processes the computation based on the operator implementation.
  1. The preferred input data type is float16 and the output data type is float32.
  2. If the float16 input data and float32 output data types are not supported, set both the input and output data types to float32.
  3. If the float32 input and output data types are not supported, set both the input and output data types to float16.
  4. If the float16 input and output data types are not supported, an error is reported.
- For vector compute operators, the operator precision in the original graph is float16 or bfloat16, this option forces the operator to use float32.
  This option takes no effect if the original graph contains operators whose implementation on the AI Core does not support float32, for example, an operator that supports only float16. In this case, the supported float16 is used. If the operator implementation on the AI Core does not support float32 and the blocklist mode is enabled (by setting precision_reduce to false), the float32 AI CPU operator is used. If the AI CPU operator does not support float32 either, an error is reported.
force_fp16:
Indicates that float16 is forcibly selected if the operator precision in the original graph is float16, bfloat16, and float32.
allow_fp32_to_fp16:
- For matrix operators:
  - If the operator computation precision in the original graph is float32, the precision is preferentially reduced to float16. If the operator does not have a float16 implementation on the AI Core, float32 is used. If the AI Core also does not support float32 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.
  - If the operator computation precision in the original graph is bfloat16, the original precision is used preferentially. If the operator does not have a bfloat16 implementation on the AI Core, float32 is used. If the AI Core also does not support float32 for this operator, the precision is reduced to float16. If the AI Core also does not support float16 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.
- For vector operators, the original precision is used preferentially.
  - If the operator computation precision in the original graph is float32, the original precision is used preferentially. If the operator does not have a float32 implementation on the AI Core, the precision is reduced to float16. If the AI Core also does not support float16 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.
  - If the operator computation precision in the original graph is bfloat16, the original precision is used preferentially. If the operator does not have a bfloat16 implementation on the AI Core, float32 is used. If the AI Core also does not support float32 for this operator, the precision is reduced to float16. If the AI Core also does not support float16 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.
must_keep_origin_dtype:
Retain the original precision.
- If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only float32 and bfloat16, the system automatically uses high-precision float32.
- If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only bfloat16, the AI CPU operator of float16 is used. If the AI CPU operator is not supported, an error is reported.
- If the precision of an operator in the original graph is float32, and the implementation of the operator in the AI Core does not support float32 but supports only float16, the AI CPU operator of float32 is used. If the AI CPU operator is not supported, an error is reported.
allow_mix_precision/allow_mix_precision_fp16:
allow_mix_precision has the same effect as that of allow_mix_precision_fp16, indicating that mixed precision of float16, bfloat16, and float32 is used for neural network processing. allow_mix_precision_fp16 is newly added to the new version, which has clearer semantics for easy understanding.

For float32 and bfloat16 operators in the original model, float16 is automatically used for certain float32 and bfloat16 operators based on the built-in tuning policy, which improves system performance and reduces memory usage with minimal accuracy loss.

If this mode is configured, you can view the value of precision_reduce in the built-in tuning policy file ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json.
- If it is set to true, the operator is on the trustlist and its precision will be reduced from float32 or bfloat16 to float16.
- If it is set to false, the operator is on the blocklist and its precision will not be reduced from float32 or bfloat16 to float16. Such operators will continue to use their original precision (float32 or bfloat16).
- If an operator in the network model does not have precision_reduce configured (that is, it is in the graylist), the mixed-precision handling mechanism for the current operator follows that of the previous operator. That is, if the previous operator supports precision reduction, the current operator also supports it; if the previous operator does not allow precision reduction, the current operator does not either.
allow_mix_precision_bf16:
Mixed precision of bfloat16 and float32 is used for neural network processing. In this mode, bfloat16 is automatically used for certain float32 operators on the original model based on the built-in tuning policy, which improves system performance and reduces memory usage with minimal accuracy loss. If the operator has neither bfloat16 nor float32 implementation on the AI Core, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.

If this mode is configured, you can view the value of precision_reduce in the built-in tuning policy file ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json.
- If it is set to true, the operator is on trustlist and its precision will be reduced from float32 to bfloat16.
- If it is set to false, the operator is on the blocklist and its precision will not be reduced from float32 to bfloat16.
- If an operator in the network model does not have precision_reduce configured (that is, it is in the graylist), the mixed-precision handling mechanism for the current operator follows that of the previous operator. That is, if the previous operator supports precision reduction, the current operator also supports it; if the previous operator does not allow precision reduction, the current operator does not either.
allow_fp32_to_bf16:
- If the operator computation precision in the original graph is float32, the original precision is used preferentially. If the operator does not have a float32 implementation on the AI Core, the precision is reduced to bfloat16. If the AI Core also does not support bfloat16 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.
- If the operator computation precision in the original graph is bfloat16, the original precision is used preferentially. If the operator does not have a bfloat16 implementation on the AI Core, float32 is used. If the AI Core also does not support float32 for this operator, the AI CPU operator is used for computation. If the AI CPU operator is also unsupported, an error is reported.

Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann. xxx in the preceding path based on the actual product.

Default value:

In the Atlas training product training scenario, the default value of this parameter is allow_fp32_to_fp16.

In the Atlas A2 training product/Atlas A2 inference product training scenario, the default value of this parameter is must_keep_origin_dtype.

In the Atlas A3 training product/Atlas A3 inference product training scenario, the default value of this parameter is must_keep_origin_dtype.

In the Atlas 350 Accelerator Card training scenario, the default value of this parameter is must_keep_origin_dtype.

In the online inference scenario, the default value of this parameter is force_fp16.

Restrictions:

The bfloat16 data type supports only the following products:

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Ascend 950PR/Ascend 950DT

Configuration example:

{"ge.exec.precision_mode", "force_fp16"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.precision_mode_v2

Sets the operator precision mode. The value must be of the string type. This parameter cannot be used together with ge.exec.precision_mode. You are advised to use ge.exec.precision_mode_v2.

Parameter values:

fp16:
Indicates that float16 is forcibly selected if the operator precision in the original graph is float16, bfloat16, or float32.
origin:
Retain the original precision.
- If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only float32 and bfloat16, the system automatically uses high-precision float32.
- If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only bfloat16, the AI CPU operator of float16 is used. If the AI CPU operator is not supported, an error is reported.
- If the precision of an operator in the original graph is float32, and the implementation of the operator in the AI Core does not support float32 but supports only float16, the AI CPU operator of float32 is used. If the AI CPU operator is not supported, an error is reported.
cube_fp16in_fp32out:
Selects different processing modes depending on the operator type when an operator supports both the float32 and float16 data types.
- For cube operators, the system processes the computation based on the operator implementation.
  1. The preferred input data type is float16 and the output data type is float32.
  2. If the float16 input data and float32 output data types are not supported, set both the input and output data types to float32.
  3. If the float32 input and output data types are not supported, set both the input and output data types to float16.
  4. If the float16 input and output data types are not supported, an error is reported.
- Forces vector compute operators in the original graph to use float32, regardless of whether their original computation precision is float16 or bfloat16.
  This option takes no effect if the original graph contains operators whose implementation on the AI Core does not support float32, for example, an operator that supports only float16. In this case, the supported float16 is used. If the operator implementation on the AI Core does not support float32 and the blocklist mode is enabled (by setting precision_reduce to false), the float32 AI CPU operator is used. If the AI CPU operator does not support float32 either, an error is reported.
mixed_float16:
Mixed precision of float16, bfloat16, and float32 is used for neural network processing. For float32 and bfloat16 operators in the original graph, float16 is automatically used for certain float32 and bfloat16 operators based on the built-in tuning policy, which improves system performance and reduces memory usage with minimal accuracy loss.

If this mode is configured, you can view the value of precision_reduce in the built-in tuning policy file ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json.
- If it is set to true, the operator is on the trustlist and its precision will be reduced from float32 or bfloat16 to float16.
- If it is set to false, the operator is on the blocklist and its precision will not be reduced from float32 or bfloat16 to float16. Such operators will continue to use their original precision (float32 or bfloat16).
- If an operator in the network model does not have precision_reduce configured (that is, it is on the graylist), the mixed-precision handling mechanism for the current operator follows that of the previous operator. That is, if the previous operator supports precision reduction, the current operator also supports it; if the previous operator does not allow precision reduction, the current operator does not either.
mixed_bfloat16:
Mixed precision of bfloat16 and float32 is used for neural network processing. In this mode, bfloat16 is automatically used for certain float32 operators in the original graph based on the built-in tuning policy, which improves system performance and reduces memory usage with minimal accuracy loss. If an operator supports neither bfloat16 nor float32, the AI CPU operator is used. If the AI CPU operator is also unsupported, an error is reported.

If this mode is configured, you can view the value of precision_reduce in the built-in tuning policy file ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json.
- If it is set to true, the operator is on trustlist and its precision will be reduced from float32 to bfloat16.
- If it is set to false, the operator is on the blocklist and its precision will not be reduced from float32 to bfloat16.
- If an operator in the network model does not have precision_reduce configured (that is, it is in the graylist), the mixed-precision handling mechanism for the current operator follows that of the previous operator. That is, if the previous operator supports precision reduction, the current operator also supports it; if the previous operator does not allow precision reduction, the current operator does not either.
mixed_hif8:
Enables automatic mixed precision, indicating that hifloat8 (for details about this data type, click here), float16, bfloat16, and float32 are used together for neural network processing. In this mode, hifloat8 is automatically used for certain float16, bfloat16, and float32 operators in the original graph based on the built-in tuning policy, which improves system performance and reduces memory usage with minimal accuracy loss.

If this mode is configured, you can view the values of precision_reduce in the built-in tuning policy file ${INSTALL_DIR}/opp/built-in/op_impl/ai_core/tbe/config/xxx/aic-xxx-ops-info-*.json.
- If it is set to true, the operator is on the trustlist and its precision will be reduced from float16, bfloat16, or float32 to hifloat8.
- If it is set to false, the operator is on the blocklist and its precision will not be reduced from float16, bfloat16, or float32 to hifloat8. In this case, the operator retains its original precision.
- If an operator in the original graph does not have precision_reduce configured (that is, it is on the graylist), the mixed-precision handling mechanism for the current operator follows that of the previous operator. That is, if the previous operator supports precision reduction, the current operator also supports it; if the previous operator does not allow precision reduction, the current operator does not either.
cube_hif8:
Forces Cube operators that support both hifloat8 and float16, bfloat16, or float32 in the original graph to use hifloat8.

Default value:

In the Atlas training product training scenario, this parameter has no default value. In this case, allow_fp32_to_fp16 (the default value of precision_mode) is used.

In the Atlas A2 training product/Atlas A2 inference product training scenario, the default value of this parameter is origin.

In the Atlas A3 training product/Atlas A3 inference product training scenario, the default value of this parameter is origin.

In the Atlas 350 Accelerator Card training scenario, the default value of this parameter is origin.

In the online inference scenario, the default value of this parameter is fp16.

Restrictions:

The bfloat16 data type supports only the following products:

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Ascend 950PR/Ascend 950DT

Configuration example:

{"ge.exec.precision_mode_v2", "fp16"};

Mandatory/Optional: optional

Effective level: global, session, and graph

Parent topic: Options