--keep_dtype

Applicability

Description

Specifies whether the data type of a specific operator in the original model remains unchanged during model build by using a configuration file.

In the inference scenario, --precision_mode or --precision_mode_v2 can be used to set the global precision mode of a network model, but it may result in performance or accuracy problems on particular operators. Therefore, you can use --keep_dtype to keep the computation precision of these operators unchanged during the build of the original network model. If the computation precision is not supported by the Ascend AI Processor, the system automatically uses the high precision supported by the operators for computation.

See Also

• Use this option in conjunction with --precision_mode or --precision_mode_v2. However, --keep_dtype does not take effect when --precision_mode is set to must_keep_origin_dtype or --precision_mode_v2 is set to origin.
• Both --customize_dtypes and --keep_dtype can be used to set the computation precision of operators. If the model inference precision needs to be improved, you are advised to use --keep_dtype to maintain the precision of the original graph. If the precision still cannot be improved, you can use --customize_dtypes to customize the computation precision of one or more operators.

  Note that if --customize_dtypes is used and the operator name is configured, the operator name may change due to operations such as fusion and splitting during internal model tuning. As a result, the configuration does not take effect and the precision is not improved. In this case, you need to obtain logs to locate the fault. For details about logs, see Log Reference.

• If both --customize_dtypes and --keep_dtype are used, the precision set by --customize_dtypes takes precedence.

Argument

Argument: Directory (including the file name) of the operator configuration file, which lists operator names or operator types. Each operator must be in a separate line.

Restrictions: To specify optypes, follow the OpType::typeName format. Put each optype in a separate line. Only optypes defined by Ascend IRs are supported. For details about how to view the optype, see How Do I Determine the Mapping Between Operators in the Original Network Model and Operators Supported by Ascend AI Processors?.

Format: The directory (including the file name) can contain letters, digits, underscores (_), hyphens (-), periods (.), and Chinese characters.

Suggestions and Benefits

None

Example

• To specify operator names in the configuration file, use the following template (named exceptionlist.cfg).

  Opname1
  Opname2
  ...

• To specify operator types in the configuration file, use the following template (named exceptionlist.cfg).

  OpType::TypeName1
  OpType::TypeName2
  ...

The Relu operator in the TensorFlow ResNet50 network model is used as an example. The operator type defined by Ascend IR is Relu.

# Set the operator name.
fp32_vars/Relu
# Set the operator type.
OpType::Relu

Upload your exceptionlist.cfg file to any directory (for example, $HOME) on the server where ATC is located. The following is an example:

--keep_dtype=$HOME/exceptionlist.cfg --precision_mode=force_fp16

At build time, the precision of the operators specified in the exceptionlist.cfg file will be preserved while those not specified in this file will be processed according to the precision mode specified by --precision_mode or --precision_mode_v2.

Dependencies and Restrictions

None