GPU/CPU Data Dump
- Install the dump tool dependencies.
pip3 install gnureadline pexpect
- Modify the training script and insert the dump configuration.
- Training configuration example in session.run mode:
import precision_tool.tf_config as npu_tf_config sess = npu_tf_config.sess_dump(sess=sess)
- Training configuration example in Estimator mode:
import precision_tool.tf_config as npu_tf_config estim_specs = tf.estimator.EstimatorSpec(training_hooks=[npu_tf_config.estimator_dump()])
- In session.run mode, the dump configuration and Rec SDK TensorFlow model saving cannot be used at the same time.
- During multi-device training, you only need to add the dump configuration to the training of one device. Otherwise, data conflicts will occur when multiple devices save data at the same time.
- Training configuration example in session.run mode:
- Perform training.
After the maximum number of training steps is changed to 1 and the training is performed, the dump data is generated in the precision_data/tf/tf_debug/ directory.
- Parse the dump data.
After python3 precision_tool/cli.py tf_dump is executed, the parsed dump data is generated in the precision_data/tf/dump/ directory. To regenerate dump data, delete the generated data and perform training and parsing again.
Parent topic: Comparison Between the GPU/CPU and NPU Networks