Deploying and Monitoring Models (MLOps basics)

The first model I deployed for a client lived on my own laptop, scored a CSV every Monday, and stopped the week I went on leave. Nobody noticed for a month. A model in a notebook creates no value; it creates value when a retention agent sees a score in her CRM at 8 a.m., or a payment is checked for fraud in 200 milliseconds, and keeps creating value after the person who built it has moved on. Getting there and staying there is called MLOps. You don't need Kubernetes. You need a repeatable pipeline, versioning, a serving choice that fits the decision, and monitoring with a retraining plan.

From notebook to pipeline

Stage	Minimum viable version	What to store
Training job	one Python script, run by cron or a scheduler	code version (git hash), data snapshot or query + date, random seed
Model registry	a folder or MLflow with model files named by version	model file, metrics on validation and test, feature list, training date
Serving	batch CSV to the database, or a small FastAPI endpoint	request logs with inputs, outputs and model version
Monitoring	a weekly SQL report and a dashboard	input distributions, prediction distribution, live accuracy when labels arrive

The one rule we'd tattoo on every graduate is that the same preprocessing code runs in training and serving. A scikit-learn Pipeline saved as one object (with joblib) guarantees the scaler, encoder and model travel together. Most "the model works in the notebook but not in production" tickets we've debugged were a scaler fitted on different data.

Batch or real time?

	Batch scoring	Real-time API
How	nightly job writes scores for all customers to a table	app calls POST /predict per event
Latency	hours	milliseconds
Fits	churn lists, demand forecasts, credit reviews	fraud at payment time, dynamic pricing, live recommendations
Complexity	low; a SQL table is the interface	higher: uptime, scaling, feature freshness

Start with batch. Move to real time only when the decision happens inside a live transaction, and be honest that the second option roughly triples the engineering.

Why models rot: drift

• Data drift: the inputs change shape. Inflation doubles basket sizes; a new city launches; a form field is renamed and arrives empty.
• Concept drift: the relationship changes. Fraudsters adapt; a competitor's promotion changes who churns.
• Label delay: you often learn the truth weeks later (a loan defaults after 90 days), so accuracy monitoring lags and input monitoring must fill the gap.

Catch input drift with PSI

The Population Stability Index compares the distribution of a feature at training time with its distribution today. For each bin, take (actual% - expected%) x ln(actual% / expected%). The feature here is transaction amount, in three bins.

Bin	Training share (expected)	This month (actual)	Difference	ln(actual / expected)	Contribution
under 5,000	0.50	0.40	-0.10	ln(0.8) = -0.223	0.0223
5,000 to 20,000	0.30	0.30	0.00	0	0.0000
over 20,000	0.20	0.30	+0.10	ln(1.5) = 0.405	0.0405
PSI					0.063

The usual rule of thumb is that PSI under 0.10 is stable, 0.10 to 0.25 is worth a look, and above 0.25 the feature has shifted enough that retraining is probably due. Here 0.063 is a mild drift toward larger amounts, consistent with inflation. Log it and watch it. Run PSI weekly on every feature and on the prediction scores themselves, because the scores drifting is often the first sign anyone gets.

Retraining and rollback

• Retrain on a schedule (monthly) or on a trigger (PSI above 0.25, live F1 below a floor).
• Always compare the candidate against the current model on the same recent window before promoting it.
• Keep the previous version deployable; a one-line rollback beats a midnight debugging session.
• Shadow mode (new model scores but does not act) or a small A/B slice de-risks the switch.

# run locally: save a full pipeline and serve it with FastAPI (pip install fastapi uvicorn joblib)
import joblib
joblib.dump(model_pipeline, "churn_v1.3.joblib")

# serve.py
from fastapi import FastAPI
import joblib, pandas as pd
app = FastAPI()
model = joblib.load("churn_v1.3.joblib")

@app.post("/predict")
def predict(row: dict):
    X = pd.DataFrame([row])
    p = float(model.predict_proba(X)[0, 1])
    return {"model_version": "1.3", "churn_probability": round(p, 4)}
# run: uvicorn serve:app --port 8000

"All the alerts are boring now": A Karachi bank deployed a card-fraud model in January. By June the investigators complained that alerts were all the same dull cases. Monitoring showed live precision had dropped from 12% to 6% while PSI on "merchant category" sat at 0.31. A wave of new online merchants had appeared that didn't exist in the training data, and the fraudsters had moved to them. The fix was a monthly retraining job on a rolling 12-month window, PSI alerts per feature, and a rule that a new model must beat the old one on the latest four weeks before promotion. The team also added a "days since model trained" tile to the dashboard, which turned out to be the single number managers looked at most.

If you remember one thing

Deploy the whole pipeline, preprocessing plus model, as one versioned artifact, and version code, data snapshot and model together so any prediction can be reproduced. Start with batch scoring and go real-time only for in-transaction decisions. Monitor the inputs (PSI), the outputs, and live accuracy once labels arrive, because every model decays. Retrain on a schedule or a trigger, compare before you promote, and keep the previous version one command away.

Homework

1. Compute PSI for a feature with training shares 0.60 / 0.40 and current shares 0.45 / 0.55. Is it stable?
2. For each, choose batch or real time and justify in one sentence: weekly churn list for a telco; fraud check inside a JazzCash transfer; monthly credit-limit review; product recommendations on a Daraz product page.
3. A model's live F1 has dropped, but every feature's PSI is under 0.05. Which kind of drift is this most likely to be, and what would you check?