DAY 17-100 DAYS MLCODE: DECISION TREES – Part 2

This blog is continuation of the previous blog about decision trees. We are going to study more about decision trees. In the last blog, we discussed the tree generated by our model ( shown below):

In the above image, Sample attribute of node means a total number of training instances [len(X)= 178] on which the algorithm was applied. A value attribute of node explains the number of training instance per class. In our first node, we have value [59, 71, 48]. That means our model was trained on 59 training of class 0 Instance, 71 of class 1 and 48 of class type 2 training instance. Lastly, Gini attributes of the node of the tree indicate the impurity. If Gini value is 0.0 means its pure node of that particular class. This means all the training set of that instance belong to a particular class. In the first node value of the Gini is 0.658 while lowest level nodes have the value of gini as 0.

To compute gini impurity for a set of items with J classes, suppose i is set {1,2,3,…J} and and let p_{i} be the fraction of items labeled with class i in the set is:

Estimating Class probabilities:

Decision tree helps to provide the probability that a particular training or test instance belong to a particular class. Let’s check the probability of one of the training instance of our model

tree_clf.predict_proba([[0.42e+01, 0.78e+00, 2.14e+00, 1.12e+01, 1.00e+02, 2.65e+00,
       5.76e+00, 5.60e-01, 8.28e+00, 5.38e+00, 1.05e+00, 3.40e+00,
       0.0e+03]])

Output: array([[0., 1., 0.]])

SciKit-Learn uses the classification and regression tree (CART) to train Decision Tree. This is also called growing tree. 

By default Gini Impurity is used but we can change it to Entropy if we wish to use Entropy impurity. This can be achieved by setting the value of parameter “criterion” as “entropy”.  A node has the value of entropy as zero when all the training instance belong to same class. Entropy impurity is define as:

Regularization:

As we know that overfitting problem of the training data can be avoided by using regularization. We can regularize the model by setting the value of hyper-parameter “max_depth”.  By default the value of this hyper-parameter is None, means unlimited. We can reduce to minimize the the overfitting issue.

There are other parameters which we can tune like min_samples_split, min_samples_leaf, max_leaf_nodes, max_features and min_weight_fraction_leaf. General guideline , reduce the Max* parameters and increase min* parameters.

In conclusion, we have discussed todays about the decision trees classifier and how to read the the decision tree created by model. We also look through the gini impurity and entropy impurity.  At end of the blog, we discussed about regularization of Decision Trees. You can find the code here.