How Do I Determine fp_point and bp_point?

fp_point and bp_point normally refer to forward propagation and backward propagation, respectively. Here, however, they indicate the first operator in forward propagation and the last operator in backward propagation. When you need to analyze a training job with the collected training iteration traces, input the key-point operators in forward and backward propagation to obtain the time taken by forward and backward propagation.

Obtain fp_point and bp_point by taking the following steps:

Outputs are usually saved as checkpoint and meta files during training, while analyzing the forward and backward propagation operators requires the graph.pbtxt file. In this process, tf.io.write_graph is used to save computational graphs. If you start training with Estimator, you can also configure model_dir in RunConfig to print the computational graph files.

Start by opening the graph.pbtxt file.

Each node indicates an operator. You need to observe the op field.

To obtain fp_point, start search from the top until the first compute operator is found. In search, type nodes and storage nodes such as Const, VariableV2, IteratorV2, Identity, Reshape, and Case, or those whose name fields contain step, Dataset, seed, and kernel should be excluded. See the following example.

In line 1732, the first compute operator appears, with MatMul in the op field, indicating a compute operator for matrix multiplication. The value of fp_point is the name of the operator, that is, dense/MatMul.

To obtain bp_point, start search from the bottom. The first node whose name includes gradients is regarded as a bp_point operator. Nodes with Assign, Const, or NoOp in the op field should be excluded. See the following example.

In line 4457, the last operator in backward propagation appears, with MatMul in the op field and the name field contains gradients. So bp_point is determined to gradients/dense/MatMul_grad/MatMul_1.

The searched operator may have been fused or renamed. To solve this problem, look up the name of the operator in the ge_proto_xxxxx_Build.txt graph generated by GE. If an exact match is found, the operator name can be used directly. If a fuzzy match is found (generally with a _1 suffix), the operator name in GE should be used.