Automatic Graph Modification

Usage Process

This part describes how to use the automatic graph modification mode for training. For details, see Figure 1.

Figure 1 Process of automatic graph modification

The normal training process includes data processing, sparse table creation, table lookup, and training startup. After automatic image modification is enabled, the training process remains unchanged, but you need to set modify_graph to True in the table lookup API and call the automatic graph modification API before training. These two steps are key steps.

Key Steps

1. Query the sparse feature table.

   Call sparse_lookup and set modify_graph to True to enable the automatic graph modification mode during table query. The default value of this parameter is False.

2. Call the automatic graph modification API.

   The required API is modify_graph_and_start_emb_cache, but there is a difference between tf.Session Training Mode and NPUEstimator Training Mode when this API is called.

   1. TF.Session Training Mode

      In tf.Session training mode, modify_graph_and_start_emb_cache needs to be explicitly called. In addition, sess.run(iterator.initializer) needs to be changed to the dataset initialization API sess.run(get_initializer(True)) or sess.run(get_initializer(False)) for automatic graph modification. The former is used for training, and the latter is used for evaluation.

   2. NPUEstimator Training Mode

      In NPUEstimator mode, you need to add GraphModifierHook of the automatic graph modification function to multiple NPUEstimator modes (train, predict, and train_and_evaluate). For example, if the current mode is train, add GraphModifierHook to the training hook to complete training in automatic graph modification mode.

Sample Code

1. Query the sparse feature table.

   from mx_rec.core.embedding import sparse_lookup
   
   embedding = sparse_lookup(hash_table, feature, send_count, dim=None, is_train=is_train,
                             access_and_evict_config=access_and_evict_config,
                             name=hash_table.table_name + "_lookup", modify_graph=True, batch=batch)

2. Call the graph modification API.
   1. tf.Session training mode:

      from mx_rec.util.initialize import get_initializer
      from mx_rec.graph.modifier import modify_graph_and_start_emb_cache
      
      if MODIFY_GRAPH_FLAG:
          logging.info("start to modifying graph")
          modify_graph_and_start_emb_cache(dump_graph=True)
      else:
          start_asc_pipeline()
      
      # train
      with tf.compat.v1.Session(config=sess_config()) as sess:
          if MODIFY_GRAPH_FLAG:
              sess.run(get_initializer(True))
          else:
              sess.run(train_iterator.initializer)
      
      # eval
      def evaluate():
          if MODIFY_GRAPH_FLAG:
              sess.run(get_initializer(False))
          else:
              sess.run(eval_iterator.initializer)

   2. NPUEstimator training mode:

      from mx_rec.graph.modifier import GraphModifierHook
      
      est.train(input_fn=lambda: input_fn(), hooks=[GraphModifierHook()])   # est is the created NPUEstimator object.