DAY 43-100 DAYS MLCODE: DNN Basic Classification
In the previous blog we finished the basics of the Deep Neural network using the TensorFlow and in this blog, we’ll implement the DNN Basic classification using TensorFlow.
We’ll use the MNIST Fashion data and we’ll train the model to predict the first 5 types of cloths.
Let’s use library of TensorFlow to get the MNIST Fashion data
import tensorflow as tf
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
Convert the data into 2d Array:
import numpy as np
nsamples, nx, ny = X_train.shape
d2_X_train = np.reshape(X_train ,(nsamples,nx*ny))
nsamples, nx, ny = X_test.shape
d2_X_test = np.reshape(X_test, (nsamples,nx*ny))
Let’s use the SciKit-Learn to scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_std = scaler.fit_transform(d2_X_train)
X_test_std = scaler.fit_transform(d2_X_test)
Prepare Train, Validation and Test data
y_train = y_train.astype(np.int32)
y_test = y_test.astype(np.int32)
X_valid, X_train_1 = d2_X_train[:5000], d2_X_train[5000:]
y_valid, y_train_1 = y_train[:5000], y_train[5000:]
Now, consider only the class 0 to 4 as a train, test and validation data as we are going to train only for five classes
X_train_5 = X_train_1[y_train_1 < 5] y_train_5= y_train_1[y_train_1 < 5] X_valid_5 = X_valid[y_valid < 5] y_valid_5= y_valid[y_valid < 5] X_test_5 = d2_X_test[y_test < 5] y_test_5 = y_test[y_test < 5]
Now, our data is standardize and in correct shape . Now time to construct the model.
Let’s build a Model with five hidden layers of 100 neurons each, He initialization, and the ELU activation function.
Instead of writing the Layers five times, let’s create a routine to create the layers
he_initialization = tf.variance_scaling_initializer()
def mnist_fashion_dnn(inputs, n_hidden_layers=5, n_neurons=100, name=None,
activation=tf.nn.elu, initializer=he_initialization):
with tf.variable_scope(name, “mnist_fashion_dnn”):
for layer in range(n_hidden_layers):
inputs = tf.layers.dense(inputs, n_neurons, activation=activation,
kernel_initializer=initializer,
name=”hidden%d” % (layer + 1))
return inputs
Let’s start the construction of the model
n_inputs = 28 * 28 # MNIST Fashion data size
n_outputs = 5 #Output layers
tf.reset_default_graph() #Reset the graph if there is any
#Create place holder X and y
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name=”X”)
y = tf.placeholder(tf.int32, shape=(None), name=”y”)
#Create Layes by calling the subroutine
inner_layer_outputs = mnist_fashion_dnn(X)
#Output Layers
logits = tf.layers.dense(inner_layer_outputs, n_outputs,
kernel_initializer=he_initialization, name=”logits”)
Y_proba = tf.nn.softmax(logits, name=”Y_proba”)
Let’s complete the Model (Graph) by adding the cost function, the training op, and validation checks
learning_rate = 0.01
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name=”loss”)
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss, name=”training_op”)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name=”accuracy”)
init = tf.global_variables_initializer()
saver = tf.train.Saver()
Now our model is ready for training and we can save model after every checkpoint.
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in select_batch(X_train_5, y_train_5, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid_5, y: y_valid_5})
if loss_val < best_loss: save_path = saver.save(sess, "./my_mnist_fashion_model_0_to_4.ckpt") best_loss = loss_val checks_without_progress = 0 else: checks_without_progress += 1 if checks_without_progress > max_checks_without_progress:
print(“Early stopping!”)
break
print(“{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%”.format(
epoch, loss_val, best_loss, acc_val * 100))
with tf.Session() as sess:
saver.restore(sess, “./my_mnist_fashion_model_0_to_4.ckpt”)
acc_test = accuracy.eval(feed_dict={X: X_test_5, y: y_test_5})
print(“Final test accuracy: {:.2f}%”.format(acc_test * 100))
Output
0 Validation loss: 0.361276 Best loss: 0.361276 Accuracy: 86.95%
1 Validation loss: 0.403741 Best loss: 0.361276 Accuracy: 86.63%
2 Validation loss: 1.625877 Best loss: 0.361276 Accuracy: 22.19%
3 Validation loss: 1.643150 Best loss: 0.361276 Accuracy: 20.11%
4 Validation loss: 1.669258 Best loss: 0.361276 Accuracy: 19.47%
5 Validation loss: 1.748824 Best loss: 0.361276 Accuracy: 18.24%
6 Validation loss: 1.643045 Best loss: 0.361276 Accuracy: 20.11%
7 Validation loss: 1.646077 Best loss: 0.361276 Accuracy: 22.19%
8 Validation loss: 1.634420 Best loss: 0.361276 Accuracy: 19.47%
9 Validation loss: 1.656327 Best loss: 0.361276 Accuracy: 20.11%
10 Validation loss: 1.841627 Best loss: 0.361276 Accuracy: 19.99%
11 Validation loss: 1.632550 Best loss: 0.361276 Accuracy: 22.19%
12 Validation loss: 1.666452 Best loss: 0.361276 Accuracy: 19.47%
13 Validation loss: 1.669140 Best loss: 0.361276 Accuracy: 19.47%
14 Validation loss: 1.720551 Best loss: 0.361276 Accuracy: 18.24%
15 Validation loss: 1.661050 Best loss: 0.361276 Accuracy: 22.19%
16 Validation loss: 1.680084 Best loss: 0.361276 Accuracy: 18.24%
17 Validation loss: 1.646257 Best loss: 0.361276 Accuracy: 18.24%
18 Validation loss: 1.612820 Best loss: 0.361276 Accuracy: 19.99%
19 Validation loss: 1.718421 Best loss: 0.361276 Accuracy: 20.11%
20 Validation loss: 1.760554 Best loss: 0.361276 Accuracy: 19.47%
Early stopping!
INFO:tensorflow:Restoring parameters from ./my_mnist_model_0_to_4.ckpt
Final test accuracy: 86.16%
We got the accuracy of 84 % on test data. we can now tune the Hyper parameters to get better output. You can find