Custom WarmStart

Usage Guide

To use the custom WarmStart function in the Rec SDK TensorFlow training framework, you need to create a WarmStart configuration tf.estimator.WarmStartSettings before creating an NPUEstimator object in the model code, and then transfer the configuration to the warm_start_from parameter in NPUEstimator.

Custom WarmStart is only supported in on-chip memory and DDR modes under TensorFlow 1.15.0.

Sample

The following is an example of customizing WarmStart:

Example 1: The model loads the sparse table user_table from the warm_start path.

import tensorflow as tf
from tf_adapter import NPUEstimator
 
warm_settings=tf.estimator.WarmStartSettings(ckpt_to_initialize_from="./warm_start",vars_to_warm_start ="user_table", var_name_to_vocab_info=None, var_name_to_prev_var_name=None)
est = NPUEstimator(
        model_fn=get_model_fn(create_fs_params, cfg, access_and_evict),
        params=params,
        model_dir=params.model_dir,
        config=run_config,
        warm_start_from=warm_settings 
)

**Table 1** tf.estimator.WarmStartSettings parameters
Parameter	Type	Description
ckpt_to_initialize_from	str(path)	Checkpoint from which initialization starts.
vars_to_warm_start	str/regular expression/list[str]/list[variables]	Variables from which initialization starts. The method of specifying variables at the dense layer is the same as that of TensorFlow. The embedding parameter supports regular expressions, *str(table name), and list(table name list)*.
var_name_to_vocab_info	dict	Vocabulary table information, which is used to restore the embedding matrix.
var_name_to_prev_var_name	dict	Stores the mapping between variable names and variable names in the WarmStart path. NOTE: The embedding table name does not support name mapping.

Example 2: The model loads all parameters from the warm_start_1 path, and then loads the embedding tables user_table and item_table from the warm_start_2 path to replace the sparse table result in the warm_start_1 path. The model loads the mlp_layer_w parameter from the warm_start_3 path to replace the loading result of warm_start_1.

import tensorflow as tf
from tf_adapter import NPUEstimator
 
ckpt_to_initialize_from_list = ["./warm_start_1", "./warm_start_2", "./warm_start_3"]
vars_to_warm_start_list=[".*",  ["user_table", "item_table"], "mlp_layer_w" ]
var_name_to_prev_var_name_list = [{}, {}, {}]
warm_settings=tf.estimator.WarmStartSettings(
        ckpt_to_initialize_from=ckpt_to_initialize_from_list,
        vars_to_warm_start = vars_to_warm_start_list,
        var_name_to_vocab_info=None,
        var_name_to_prev_var_name=var_name_to_prev_var_name_list )
 
 est = NPUEstimator(
        model_fn=get_model_fn(create_fs_params, config, access_and_evict),
        params=params,
        model_dir=params.model_dir,
        config=run_config,
        warm_start_from=warm_settings
    )

**Table 2** **tf.estimator.WarmStartSettings** parameters for the multi-path warm start function
Parameter	Type	Description
ckpt_to_initialize_from	List(str(path))	Checkpoint from which initialization starts.
vars_to_warm_start	List (str/regular expression/list[str]/list[variables])	Variables from which initialization starts. The method of specifying variables at the dense layer is the same as that of TensorFlow. The embedding parameter supports regular expressions, *str(table name), and list(table name list)*.
var_name_to_vocab_info	List(dict)	Vocabulary table information, which is used to restore the embedding matrix.
var_name_to_prev_var_name	List(dict)	Stores the mapping between variable names and variable names in the WarmStart path. The embedding table name does not support name mapping.

Parent topic: Function Training Process