Decision Trees and Random Forests

Show a loan officer a decision tree and she'll say "that's how I think anyway". Ask a question, branch, ask another, decide. Trees handle mixed data, need no scaling, and fit on a whiteboard. Their weakness is that a single tree overfits at the first opportunity, which is where the random forest comes in: hundreds of slightly different trees that vote. Between them they're the most common "first serious model" we see in Pakistani companies.

How a tree makes a decision

Each internal node tests one feature against a threshold. A new applicant starts at the root and follows the yes/no branches down to a leaf. The leaf's majority class (or, for regression, the mean of its training rows) is the prediction. The leaf's class proportions also give you a probability: 80 paid out of 85 in a leaf means p(paid) = 0.94.

How the tree chooses its questions

Training a tree means picking, at each node, the feature and threshold that best separate the classes. "Best" is measured by impurity, and the usual measure is Gini impurity, 1 - sum of (class proportion)^2. A pure node (all one class) has Gini 0; a 50/50 node has Gini 0.5.

One split, computed by hand

The root node holds 200 applicants, 120 paid and 80 defaulted.

Gini(root) = 1 - (0.60^2 + 0.40^2) = 1 - (0.36 + 0.16) = 0.48.

Candidate A: income > 60,000?

Branch	Rows	Paid	Default	Gini	Weight
no	90	30	60	1 - (0.333^2 + 0.667^2) = 0.444	90 / 200 = 0.45
yes	110	90	20	1 - (0.818^2 + 0.182^2) = 0.298	110 / 200 = 0.55

Weighted Gini after split = 0.45 x 0.444 + 0.55 x 0.298 = 0.200 + 0.164 = 0.364. Gain = 0.48 - 0.364 = 0.116.

Candidate B: late payments > 2?

Branch	Rows	Paid	Default	Gini	Weight
yes	60	25	35	1 - (0.417^2 + 0.583^2) = 0.486	0.30
no	140	95	45	1 - (0.679^2 + 0.321^2) = 0.436	0.70

Weighted Gini = 0.30 x 0.486 + 0.70 x 0.436 = 0.146 + 0.305 = 0.451. Gain = 0.029.

Candidate A cuts impurity far more, so the root asks about income, exactly as the diagram shows. The algorithm then repeats the same search inside each branch until leaves are pure or a stopping rule (max_depth, min_samples_leaf) kicks in. That's the whole training procedure, and it's a favourite interview question: "walk me through how a tree picks its first split".

Why one tree overfits and how to control it

Setting	Effect	Typical values
max_depth	limits the number of questions in a row	3 to 8
min_samples_leaf	a leaf must hold at least this many rows	20 to 100 for business data
min_impurity_decrease	ignore splits with tiny gains	0.001 to 0.01

Leave a tree unlimited and it keeps splitting until every leaf is pure, which means it has memorised the noise. Two nearly identical datasets can produce completely different trees. That instability is what "high variance" means in practice.

Random forest: many trees, one vote

A random forest grows N trees (typically 100 to 500), each on a bootstrap sample (rows drawn with replacement) and, at every split, considering only a random subset of features. Both kinds of randomness make the trees disagree, and averaging many disagreeing but individually decent models cancels their errors. Variance drops sharply while bias stays low. The forest also gives you feature importance (how much each feature reduced impurity across all trees) and an out-of-bag score computed on the rows each tree never saw, which is a free validation estimate.

In the diagram three of four trees vote churn, so the forest predicts churn with probability 0.75. With 300 trees that probability becomes a smooth ranking score you can actually sort a call list by.

# run locally
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.ensemble import RandomForestClassifier

tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=20).fit(X_train, y_train)
print(export_text(tree, feature_names=list(X_train.columns)))   # the whiteboard rules

rf = RandomForestClassifier(n_estimators=300, max_features="sqrt",
                            min_samples_leaf=10, oob_score=True,
                            random_state=0, n_jobs=-1).fit(X_train, y_train)
print("OOB accuracy:", rf.oob_score_)
for name, imp in sorted(zip(X_train.columns, rf.feature_importances_),
                        key=lambda t: -t[1])[:5]:
    print(name, round(imp, 3))

Tree vs forest at a glance

	Single tree	Random forest
Explainability	excellent, draw it	good via importances, not per decision
Accuracy	moderate	strong out of the box
Overfitting	easy	hard
Training time	seconds	minutes, parallel
Needs scaling / one-hot	no / integers tolerated	no / integers tolerated

A tree on the staffroom wall: A Rawalpindi school chain wanted to spot students at risk of failing the matric board exam early in the year. A depth-3 tree on attendance, mid-term average, homework completion and previous-year grade produced rules teachers accepted on sight, such as "attendance below 75% and mid-term below 45 puts a student in the high-risk leaf, where 68% failed last year". A 300-tree forest on the same features lifted recall of at-risk students from 61% to 74%. The school ranked students for tutoring slots with the forest and printed the tree on the staffroom wall to explain why.

Before you move on

• A tree splits on the feature and threshold that reduce Gini impurity the most, then repeats inside each branch.
• Leaves give both a class and a probability from their training proportions.
• Unlimited trees memorise. Control depth and leaf size.
• Random forests average many randomised trees: low variance, strong defaults, feature importances for free.
• Shallow tree to explain, forest to predict.

Three things to try

1. Compute the Gini impurity of a node with 70 paid and 30 defaulted applicants.
2. A third candidate split "loan / income > 8" gives branches of (50 rows: 15 paid, 35 default) and (150 rows: 105 paid, 45 default). Compute its weighted Gini and gain. Does it beat income?
3. Explain in two sentences why a random forest usually beats a single unlimited tree on new data even though the single tree has lower training error.