DAY 44-100 DAYS MLCODE: Reuse pre-trained Layers

In the previous blog, we trained the layers to detect the five type of cloths using the MNIST fashion data, in this blog, we’ll reuse pre-trained layers to create a new model to detect the class 5-9.

This process of reusing the pre-trained layers to create a model is called Transfer Learning. Let’s create a new DNN that reuses all the pre-trained hidden layers of the previous model, we’ll freezes them.

restore_saver = tf.train.import_meta_graph(“./my_mnist_model_0_to_4.ckpt.meta”)

X = tf.get_default_graph().get_tensor_by_name(“X:0”)
y = tf.get_default_graph().get_tensor_by_name(“y:0”)
loss = tf.get_default_graph().get_tensor_by_name(“loss:0”)
Y_proba = tf.get_default_graph().get_tensor_by_name(“Y_proba:0”)
logits = Y_proba.op.inputs[0]
accuracy = tf.get_default_graph().get_tensor_by_name(“accuracy:0”)

Let’s freeze the lower layers. We will exclude lower level variables from the optimizer’s list of trainable variables, keeping only the output layer’s trainable variables

learning_rate = 0.01
output_layer_variables = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=”logits”)
optimizer = tf.train.AdamOptimizer(learning_rate, name=”Adam2″)
training_op = optimizer.minimize(loss, var_list=output_layer_variables )

Let’s train this new DNN on class 5 to 9, using only 100 images per class, and time how long it takes. Despite this small number of examples, can you achieve high precision?

Split the data into test, train and validation datasets:

X_train_5_9 = X_train_1[y_train_1 >= 5]
y_train_5_9 = y_train_1[y_train_1 >= 5] – 5
X_valid_5_9 = X_valid[y_valid >= 5]
y_valid_5_9 = y_valid[y_valid >= 5] – 5
X_test_5_9 = d2_X_test[y_test >= 5]
y_test_5_9 = y_test[y_test >= 5] – 5

Select the 100 training instances and 30 validation ones

X_train_529_100, y_train_529_100 = select_n_instances_per_class(X_train_5_9, y_train_5_9, n=100)
X_valid_529_100, y_valid_529_100 = select_n_instances_per_class(X_valid_5_9, y_valid_5_9, n=30)

Now we have data so let’s train the model. This is the same code which we have used in the previous blog except for the restore parameters values.

import time

n_epochs = 1000
batch_size = 20

max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty

with tf.Session() as sess:
init.run()
restore_saver.restore(sess, “my_mnist_model_0_to_4.ckpt.meta”)
t0 = time.time()

for epoch in range(n_epochs):
for X_batch, y_batch in select_batch(X_train_529_100, y_train_529_100, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid_529_100, y: y_valid529_100})
if loss_val < best_loss: save_path = five_frozen_saver.save(sess, "./my_mnist_model_5_to_9") best_loss = loss_val checks_without_progress = 0 else: checks_without_progress += 1 if checks_without_progress > max_checks_without_progress:
print(“Early stopping!”)
break
print(“{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%”.format(
epoch, loss_val, best_loss, acc_val * 100))

t1 = time.time()
print(“Total training time: {:.1f}s”.format(t1 – t0))

with tf.Session() as sess:
five_frozen_saver.restore(sess, “./my_mnist_model_5_to_9”)
acc_test = accuracy.eval(feed_dict={X: X_test_5_9, y: y_test_5_9})
print(“Final test accuracy: {:.2f}%”.format(acc_test * 100))

Although we are using pre-trained data our accuracy is not good, and it did not help to improve the performance. You can find the code here.