Cross-Validation and Hyperparameter Tuning

"Your model scored 0.84. How sure are you?" That question, from a Lahore bank's second-round interview, sends most candidates quiet. A single train/test split gives you one number, and that number depends on which rows happened to land in the test set. Cross-validation replaces one lucky estimate with several and averages them. Hyperparameter tuning then uses those estimates to pick settings like tree depth or regularisation strength. Together they turn "it scored 0.84 once" into "it scores 0.83 plus or minus 0.02, and here's why we chose depth 6".

Parameters versus hyperparameters

	Parameters	Hyperparameters
What	numbers the algorithm learns (weights, split thresholds)	settings you choose before training (k, max_depth, C, learning_rate)
Found by	fitting on training data	searching with validation data
Examples	b0 = 20, b1 = 4	n_neighbors = 7, alpha = 0.1

k-fold cross-validation

Split the training data into k folds (5 or 10). For each fold, train on the other k - 1 and score on the one held out. Every row gets validated exactly once and you end up with k scores instead of one. Report the mean as your estimate and the standard deviation as your uncertainty. The final test set stays separate and untouched the whole time.

Reading CV output for two models

Two candidates scored on the same 5 folds (F1).

Fold	Model A: logistic	Model B: boosted trees
1	0.80	0.84
2	0.78	0.81
3	0.82	0.86
4	0.79	0.79
5	0.81	0.85
Mean	0.800	0.830
Std (population)	0.014	0.026

Model A's std comes from deviations 0, -0.02, 0.02, -0.01, 0.01; squares sum 0.001; / 5 = 0.0002; sqrt = 0.014. Model B has deviations 0.01, -0.02, 0.03, -0.04, 0.02; squares sum 0.0034; / 5 = 0.00068; sqrt = 0.026. B is better on average by 0.03 and wins on 4 of 5 folds, so the improvement is probably real. But B's spread is twice A's, so it's less stable across data subsets. Had the means been 0.80 and 0.81 with these spreads, you shouldn't call B better at all.

Variants you will need

Variant	Use when
StratifiedKFold	classification; keeps class balance in every fold (default for classifiers in scikit-learn)
GroupKFold	rows share an entity (many orders per customer); keeps a customer entirely in one fold
TimeSeriesSplit	time-ordered data; each fold trains on the past and validates on the next period
Repeated k-fold	small datasets; repeat with different shuffles for a steadier estimate

How many folds, and what does the spread mean?

Five folds is the everyday default. Each model trains on 80% of the data and you pay for five fits. Ten folds gives a slightly less biased estimate at twice the cost and is worth it on small datasets (under a few thousand rows). Leave-one-out (k = number of rows) is rarely worth it. Whatever k you pick, treat the standard deviation as a warning light. If two models' means differ by less than about one standard deviation, the data can't tell them apart, and you should take the simpler or cheaper one. A big spread across folds also usually means some fold holds a segment (a city, a month, a product line) that behaves differently. Go and find it, because that's a business insight and a modelling risk in one.

Hyperparameter search

• Grid search tries every combination: 3 depths x 3 tree counts = 9 fits per fold, 45 with 5-fold CV. Exhaustive but explodes with more parameters (5 parameters with 5 values each = 3,125 combinations).
• Random search samples, say, 30 random combinations from ranges. It usually finds a near-best setting much faster because only a few hyperparameters matter.
• Bayesian optimisation (Optuna, scikit-optimize) uses earlier results to pick promising settings next; worth it when each fit takes minutes.

One rule above all the others: tune on the validation folds, never on the test set. Compare 50 settings on the test set, keep the best, and your test score is now optimistically biased. A nested scheme (CV inside CV) is the strict fix. In practice a separate untouched test set checked once is enough, and it's what we do on client work.

# run locally
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
                                     GridSearchCV, RandomizedSearchCV)
from sklearn.ensemble import RandomForestClassifier

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
rf = RandomForestClassifier(random_state=0, n_jobs=-1)

scores = cross_val_score(rf, X_train, y_train, cv=cv, scoring="f1")
print(scores.round(3), scores.mean().round(3), scores.std().round(3))

grid = GridSearchCV(rf, {"max_depth": [3, 6, 12],
                         "n_estimators": [50, 200, 500]},
                    cv=cv, scoring="f1", n_jobs=-1).fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))

rand = RandomizedSearchCV(rf, {"max_depth": range(2, 20),
                               "min_samples_leaf": range(1, 50),
                               "max_features": ["sqrt", 0.3, 0.6]},
                          n_iter=30, cv=cv, scoring="f1",
                          random_state=0, n_jobs=-1).fit(X_train, y_train)
print(rand.best_params_, round(rand.best_score_, 3))
# final check, once:
print("test F1:", f1_score(y_test, rand.best_estimator_.predict(X_test)))

The test set that got used up: A mobile-wallet analytics team tuned a churn model by grid-searching 120 combinations and keeping the one with the highest test-set F1, 0.71. In production it delivered 0.62. The test set had been spent by the search. They rebuilt the process with GroupKFold by customer on the training months, RandomizedSearchCV with 40 draws, best CV F1 0.66 plus or minus 0.02, then one look at a held-out month, which gave 0.65. A lower number, and one that matched production, which is what a performance estimate is for.

Before you move on

• k-fold CV gives a mean and a spread. Report both and compare models fold by fold.
• Use stratified, group or time-series splits to match how the rows are related.
• Search hyperparameters with CV; random search is usually the best value for the effort.
• Never pick settings using the test set. Open it once, at the end.
• A model that scores lower but honestly is worth more than one that scores high through leakage.

If you're preparing for interviews, "explain k-fold" and "what's the difference between a parameter and a hyperparameter" are two of the most common questions students report back to us, and we run through both in interview preparation.

Three things to try

1. Five-fold scores for a model are 0.70, 0.74, 0.69, 0.73, 0.74. Compute the mean and population std.
2. You have 4 hyperparameters with 6 candidate values each, 5-fold CV, and each fit takes 30 seconds. How long would a full grid search take? What would you do instead?
3. A Lahore hospital dataset has 3,000 visits from 800 patients. Which CV variant should you use, and why?