What Do I Do When an Error Indicating Uninitialized Variables Is Reported During Training?

Symptom

An error is reported during training, indicating that the variable operator is not initialized.

Possible Cause

During data preprocessing, tf.keras.backend.zero is used to generate variables. The variable operator cannot be executed on the device side and thereby fails to be initialized.

Solution

Modify the training script. Use the native TensorFlow API tf.zero instead of tf.keras.backend.zero to generate variables on the host in tensor mode.

Original script

1
2
3
4

y = {
   'mlm_loss': tf.keras.backend.zero([1]),
   'mlm_acc': tf.keras.backend.zero([1]),
}

Script after modification

1
2
3
4

y = {
   'mlm_loss': tf.zero([1]),
   'mlm_acc': tf.zero([1]),
}