Day 12-100 Days MLCode: Logistic Regression

One of the famous tasks of supervised learning is classification and Logistic regression is one of the most famous models for classification problems.  Logistic Regression is used when the dependent variable(target) is categorical. Logistic regression, like Linear regression, has the cost function and we have to estimates the weight and biases using gradient descent. However, the main difference is there between them is the cost function. 

Cost function of Logistic Regression:

The cost function of the Logistic regression is:

The logistic also called logit is a sigmoid function which outputs an input to 0 or 1.  Consider our hypothesis is like below:

Z = WX + B then hΘ(x) = sigmoid (Z) looks like below

From the above chart, we can see that a Z moves towards higher, sigmoid activation will return 1 and 0 in opposite case.

We can use the logistic regression to detect the class like Spam classifier, a tumor is malignant etc.

Let’s implement a small classifier using the SciKit Learn.

We are going to download the MNIST dataset from the Sci-Kit Learn and then predict the digit. Let’s download the data and prepare the train and test data

from sklearn import datasets
mnist_data = datasets.load_digits()

Check the structure of the dataset

list(mnist_data.keys())

Output: [‘data’, ‘target’, ‘target_names’, ‘images’, ‘DESCR’]

Prepare the train and test data using Scikit Learn

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test 
  = train_test_split(mnist_data.data,(mnist_data.target == 7).astype(np.int), test_size=0.20, random_state=42)

Let’s train a model to detect there the particular digit is 7 or not. In the above code, we have modified our label data to mark digit 7 as 1 and for all other digits mark as 0.

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

Let’s predict something to see whether our model is working or not.

# Predict for One Observation (image)
import random
i = random.randint(0, (len(X_test)-1))
print(f"Prediction is {log_reg.predict(X_test[i].reshape(1,-1))} and the actual value is {y_test[i]}")

Output : Prediction is [1] and the actual value is 1.

Let’s verify the mage by displaying the data using pyplot

import matplotlib.pyplot as plt
plt.figure(figsize=(20,4))
plt.imshow(np.reshape(X_test[i], (8,8)) ,cmap=plt.cm.gray)
plt.title('Training: %i\n' % y_test[i], fontsize = 20)

Like any other Linear model, Logistic regression can be regularized by l1 or l2 penalties.

In the above example, we have created a binary classifier, i.e. whether the digit is 7 or not. We can use the Softmax Regression to model the multiclass classifier.

Softmax Regression:

Logistic Regression model can be created to support the multiclass without going for training the multiple binary models and this is called Softmax Regression. Below equation shows the Softmax regression. The difference here is with logistics that it has an extra variable k. k is the number of classes.

Softmax equation

and the cost function is :

Softmax cost function

Use the SciKit-Learn Logistic regression class to train the model. We can switch to Softmax by passing the value of multi_class as “multinomial”, by default SciKit-Learn use “one vs all ” approach. We have to use solver to support the softmax and it also applies l2 regularization which you can control by a passing value of c.

from sklearn.linear_model import LogisticRegression
softmax_reg = LogisticRegression(multi_class="multinomial", solver="lbfgs", C=10)
softmax_reg.fit(X_train, y_train)

Output: LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class=’multinomial’, n_jobs=1, penalty=’l2′, random_state=None, solver=’lbfgs’, tol=0.0001, verbose=0, warm_start=False)

Let’s test the model now:

# Predict for One Observation (image)
import random
i = random.randint(0, (len(X_test)-1))
print(f"Prediction is {softmax_reg.predict(X_test[i].reshape(1,-1))} and the actual value is {y_test[i]}")

Prediction is [5] and the actual value is 5

# Use score method to get accuracy of model
score = softmax_reg.score(X_test, y_test)
print(score)

Output : 0.9666666666666667

This shows that system is predicting approximate 97% of time correctly.

In conclusion, we have implemented Binary classifier and multiclass classifier using the simple digit data of SciKit-Learn library. Logistic is a form of regression which is used in the class detection problem of machine learning. You can find the code here and the previous day blog here.