GPU/CPU Data Dump

  1. Install the dump tool dependencies.
    pip3 install gnureadline pexpect
  2. Modify the training script and insert the dump configuration.
    • Training configuration example in session.run mode:
      import precision_tool.tf_config as npu_tf_config
      sess = npu_tf_config.sess_dump(sess=sess)
    • Training configuration example in Estimator mode:
      import precision_tool.tf_config as npu_tf_config
      estim_specs = tf.estimator.EstimatorSpec(training_hooks=[npu_tf_config.estimator_dump()])
    • In session.run mode, the dump configuration and Rec SDK TensorFlow model saving cannot be used at the same time.
    • During multi-device training, you only need to add the dump configuration to the training of one device. Otherwise, data conflicts will occur when multiple devices save data at the same time.
  3. Perform training.

    After the maximum number of training steps is changed to 1 and the training is performed, the dump data is generated in the precision_data/tf/tf_debug/ directory.

  4. Parse the dump data.

    After python3 precision_tool/cli.py tf_dump is executed, the parsed dump data is generated in the precision_data/tf/dump/ directory. To regenerate dump data, delete the generated data and perform training and parsing again.