Evaluating Models: Accuracy, Precision, Recall, F1, ROC, RMSE

A fraud model that flags nothing is 99.8% accurate, because 99.8% of transactions are honest. Read that twice. It's the reason "what accuracy did you get?" is the wrong first question, and the reason every interviewer in Karachi will ask you what precision and recall are. This lesson gives you the whole toolbox, confusion matrix, precision, recall, F1, ROC and AUC for classification and MAE, RMSE, MAPE and R^2 for regression, with every number computed by hand so you can sanity-check any library output.

Classification starts with the confusion matrix

Cell	Meaning	Fraud example (1,000 transactions, 50 fraud)
TP (true positive)	predicted fraud, was fraud	40
FN (false negative)	predicted honest, was fraud	10
FP (false positive)	predicted fraud, was honest	30
TN (true negative)	predicted honest, was honest	920

Every classification metric from those four cells

Metric	Formula	Calculation	Question it answers
Accuracy	(TP + TN) / all	960 / 1,000 = 96.0%	How often is the model right overall?
Precision	TP / (TP + FP)	40 / 70 = 57.1%	When it says fraud, how often is it right?
Recall (sensitivity)	TP / (TP + FN)	40 / 50 = 80.0%	Of all fraud, how much did we catch?
Specificity	TN / (TN + FP)	920 / 950 = 96.8%	Of all honest, how many were cleared?
F1	2 x P x R / (P + R)	2 x 0.571 x 0.8 / 1.371 = 66.7%	One number balancing precision and recall
False positive rate	FP / (FP + TN)	30 / 950 = 3.2%	How many honest customers were bothered?

Now run the "flag nothing" model through the same table. TP 0, FN 50, FP 0, TN 950. Accuracy 95%, recall 0%, precision undefined, F1 0. Four cells expose it in one glance.

Precision and recall pull in opposite directions

Lower the threshold and you flag more, so recall rises and precision falls. Raise it and the reverse happens. Which one matters depends on what each mistake costs.

Situation	Costly mistake	Optimise for
Fraud, disease screening, churn of premium customers	a miss (FN)	recall, subject to a precision floor
Blocking payments, spam filtering, auto-rejecting loans	a false alarm (FP)	precision
Balanced costs	both	F1, or a cost-weighted metric

ROC curve and AUC: judging the ranking, not one threshold

Sweep the threshold from 1 down to 0, record (false positive rate, recall) at each step, and plot the points. That's the ROC curve. The area under it (AUC) is the probability that a random positive scores higher than a random negative. 0.5 is a coin flip, 0.8 is useful, 0.9 and above is strong. AUC lets you compare models before the business has picked a threshold. When positives are very rare, also look at the precision-recall curve, which is far more sensitive to the rare class and, in our experience, far more honest on fraud data.

Regression metrics

Five days of bread forecasts for a bakery, in loaves.

Day	Actual	Predicted	Error	|Error|	Error^2	|Error| / Actual
1	100	110	-10	10	100	10.0%
2	120	115	5	5	25	4.2%
3	90	95	-5	5	25	5.6%
4	150	130	20	20	400	13.3%
5	110	110	0	0	0	0.0%
Sum			10	40	550	33.1%

• MAE = 40 / 5 = 8 loaves: typical miss, easy to explain.
• RMSE = sqrt(550 / 5) = sqrt(110) = 10.5 loaves: punishes the big miss on day 4 more; always at least MAE.
• MAPE = 33.1% / 5 = 6.6%: unit-free, but explodes when actuals are near zero.
• R^2 = 1 - SSE / SST. Mean actual = 114; SST = 196 + 36 + 576 + 1,296 + 16 = 2,120; R^2 = 1 - 550 / 2,120 = 0.74.
• Bias = mean error = 10 / 5 = +2: the model slightly under-forecasts on average.

# run locally
from sklearn.metrics import (confusion_matrix, classification_report,
                             roc_auc_score, mean_absolute_error,
                             mean_squared_error, r2_score)
import numpy as np

# classification
proba = clf.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)
print(confusion_matrix(y_test, pred))          # [[TN FP] [FN TP]]
print(classification_report(y_test, pred, digits=3))
print("AUC:", round(roc_auc_score(y_test, proba), 3))

# regression
y_hat = reg.predict(X_test)
print("MAE", mean_absolute_error(y_test, y_hat))
print("RMSE", np.sqrt(mean_squared_error(y_test, y_hat)))
print("R2", r2_score(y_test, y_hat))

The vendor with 98.9% accuracy: A Karachi bank's fraud team was shown a new model with "98.9% accuracy" by a vendor. The analyst asked for the confusion matrix on last month's 400,000 transactions with 600 confirmed frauds. It showed TP 210, FN 390, FP 3,800. Recall was 35% and precision 5%. The bank's existing rules had recall 48% at precision 7%. Accuracy had looked wonderful only because 99.85% of transactions were honest. The vendor was sent away to come back with a precision-recall curve and the expected fraud caught per 2,000 daily alerts, which was the only number the investigators cared about.

If you remember one thing

Never quote accuracy alone on imbalanced data; show the confusion matrix. Precision is how much to trust a positive call, recall is the share of positives you caught, and F1 balances them. AUC judges the ranking across all thresholds, and PR curves do the same job better when positives are rare. For regression, MAE is the typical miss, RMSE punishes big misses, MAPE is unit-free and R^2 is variance explained. Pick the metric from the business cost of each kind of mistake, and pick it before you train.

Homework

1. A churn model on 2,000 customers gives TP 150, FN 100, FP 200, TN 1,550. Compute accuracy, precision, recall and F1.
2. For the bakery table, suppose day 4's prediction is corrected to 145. Recompute MAE and RMSE. Which changed more, and why?
3. A Rawalpindi school wants to flag at-risk students for tutoring, and slots are limited to 40 out of 500 students. Which metric would you report, and how would you set the threshold?