文档

Modifying the Blocklist, Trustlist, and Graylist for Mixed Precision

Overview

When automatic mixed precision is enabled, the system automatically reduces the precisions of some data types on a network based on the built-in tiling policy. This improves the system performance while reducing the memory usage at low accuracy loss.

Find the built-in tiling policy in /opp/built-in/op_impl/ai_core/tbe/config/<soc_version>/aic-<soc_version>-ops-info.json under the CANN installation directory.
1
2
3
4
"Conv2D":{
    "precision_reduce":{
        "flag":"true"
},
  • Scenarios where precision_mode_v2 is set to mixed_float16 and precision_mode is set to allow_mix_precision_fp16/allow_mix_precision:
    • If the field value is true, the operator is on the mixed precision trustlist and its precision will be reduced from float32 to float16.
    • If the field value is false, the operator is on the mixed precision blocklist and its precision will not be reduced from float32 to float16.
    • If an operator does not have the precision_reduce option configured, the operator is on the graylist and will follow the same precision processing as the upstream operator.

You can specify operators to reduce or preserve the precision based on the built-in tiling policy.

The following describes two configuration methods.

(Recommended) Using modify_mixlist to Specify the Blocklist, Trustlist, and Graylist for Mixed Precision.

In the training script, use the modify_mixlist parameter to specify the configuration file of the blocklist, trustlist, and graylist for mixed precision. The following is a configuration example:

  • Automated porting
    1
    custom_op.parameter_map["modify_mixlist"].s = tf.compat.as_bytes("/home/test/ops_info.json")
  • Manual porting
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
    from npu_bridge.npu_init import *
# In Estimator mode
npu_config=NPURunConfig(
  model_dir=FLAGS.model_dir,
  save_checkpoints_steps=FLAGS.save_checkpoints_steps,
  session_config=tf.ConfigProto(allow_soft_placement=True,log_device_placement=False),
  precision_mode_v2="mixed_float16",
  modify_mixlist="/home/test/ops_info.json"
  )
# In sess.run mode
config = tf.ConfigProto()
custom_op =  config.graph_options.rewrite_options.custom_optimizers.add()
custom_op.name =  "NpuOptimizer" 
custom_op.parameter_map["use_off_line"].b = True
custom_op.parameter_map["precision_mode_v2"].s = tf.compat.as_bytes("mixed_float16")
custom_op.parameter_map["modify_mixlist"].s = tf.compat.as_bytes("/home/test/ops_info.json")
config.graph_options.rewrite_options.remapping = RewriterConfig.OFF
config.graph_options.rewrite_options.memory_optimization = RewriterConfig.OFF
with tf.Session(config=config) as sess:
  print(sess.run(cost))

# In Keras mode
The modification method is similar to that in sess.run mode.
ops_info.json is the configuration file of the blocklist, trustlist, and graylist for mixed precision. Multiple operators are separated by commas (,). An example is as follows:
{
  "black-list": {                  // Blocklist
     "to-remove": [                // Move an operator from the blocklist to the graylist.
     "Xlog1py"
     ],
     "to-add": [                   // Move an operator from the trustlist or graylist to the blocklist.
     "Matmul",
     "Cast"
     ]
  },
  "white-list": {                  // Trustlist
     "to-remove": [                // Move an operator from the trustlist to the graylist.
     "Conv2D"
     ],
     "to-add": [                   // Move an operator from the blocklist or graylist to the trustlist.
     "Bias"
     ]
  }
}

Assume that operator A is in the trustlist by default. If you want to move it to the blocklist, follow any of the positive examples below:

  1. (Positive example) Directly add the operator to the blocklist.
    1
2
3
4
5
    {
  "black-list": { 
     "to-add": ["A"]
  }
}

    The operator will be deleted from the trustlist and added to the blocklist. You can find it in the blocklist.

  2. (Positive example) Delete the operator from the trustlist and add it to the blocklist.
    1
2
3
4
5
6
7
8
    {
  "black-list": {
     "to-add": ["A"]
  },
  "white-list": {
     "to-remove": ["A"]
  }
}

    The operator will be deleted from the trustlist and added to the blocklist. You can find it in the blocklist.

  3. (Negative example) Simply delete the operator from the trustlist. In this case, the operator will be moved to the graylist instead of the blocklist.
    1
2
3
4
5
    {
  "white-list": {
     "to-remove": ["A"]
  }
}

    The operator will be deleted from the trustlist and added to the graylist.

    If an operator is simply removed from the blocklist or trustlist, it will be added to the graylist.

Modifying the Operator Information Library

Modifying the built-in operator information library may affect other networks. Proceed with caution.

  1. Go to /opp/built-in/op_impl/ai_core/tbe/config/<soc_version> under the CANN installation directory.
  2. Grant the write permission on the aic-<soc_version>-ops-info.json file.
    chmod u+w aic-<soc_version>-ops-info.json

    All .json files in the current directory will be loaded to the operator information library. If you need to back up the original .json files, back them up to another directory.

  3. Modify or add the precision_reduce field of the corresponding operator in the aic-<soc_version>-ops-info.json file in the operator information library.