Computer Vision in the Real World

A factory manager in Sialkot once told us his defect camera "worked perfectly in the demo and never again". The demo ran at midday. The night shift ran under yellow sodium lights. That's computer vision in one story. The models are good, and the conditions decide whether they work. Vision reads number plates on the Lahore ring road, checks CNIC photos at bank branches, counts customers in shops and spots flaws on a production line. Here are the main tasks, how a real pipeline is built, how to measure it, and the decisions an analyst ends up making.

Four core tasks

Task	Question it answers	Output	Pakistani example
Classification	What is in this image?	One label with confidence	Is this fabric roll defective or fine?
Object detection	Where are the objects?	Boxes with labels	Count motorbikes at a Karachi junction
Segmentation	Which pixels belong to what?	A mask per object or class	Measure crop area from drone images in Punjab
OCR	What does the text say?	Text strings with positions	Read CNIC, utility bills, handwritten challans

Everything else builds on those four. Face recognition is detection plus identity matching, pose estimation finds body keypoints for safety monitoring, tracking follows objects across video frames, and visual question answering is asking a multimodal model "how many items are on this shelf?"

How the models work

Most vision models are CNNs or vision transformers (lesson 8), nearly always pre-trained on millions of general images and then fine-tuned on a few hundred to a few thousand labelled images of your own. Since 2024 a multimodal LLM can do all four tasks from a plain-language prompt with no training, which is perfect for a prototype or low-volume work. For high volume (thousands of frames a minute) or anything on a device, a dedicated small model is cheaper and faster. We tell students to prototype with the multimodal model and only train something when the volume justifies it.

The production pipeline

1. Capture. Camera placement, lighting and resolution decide more of the final accuracy than the model. A fixed camera under consistent light beats a better algorithm. Ask the Sialkot manager.
2. Pre-process. Resize, crop to the region you care about, correct brightness, de-skew documents.
3. Model. Run detection, classification or OCR.
4. Post-process. Apply confidence thresholds, business rules (a CNIC number has 13 digits, full stop), and de-duplication across frames.
5. Action. Alert, write to a database, update a dashboard, or route to a person.

Measuring it

• Classification. Precision and recall per class, plus a confusion matrix showing which classes get mixed up.
• Detection. Mean average precision (mAP), which scores both where the boxes land and what they're labelled.
• OCR. Character error rate, and field-level accuracy (was the whole CNIC number right?).
• New conditions, always. Night, rain, dust on the lens, a different phone camera, a worn and faded card.

Picking a confidence threshold

A defect detector gives a probability for each fabric segment. Where you set the cut-off trades missed defects against false alarms. This plain Python works out precision and recall at three thresholds from a small set of scored segments.

scored = [  # (model probability of defect, actually defective)
    (0.95, 1), (0.90, 1), (0.85, 0), (0.80, 1), (0.70, 1),
    (0.60, 0), (0.55, 1), (0.40, 0), (0.30, 0), (0.20, 0), (0.10, 1), (0.05, 0),
]
for t in (0.5, 0.7, 0.9):
    flagged = [(p, y) for p, y in scored if p >= t]
    tp = sum(y for _, y in flagged)
    fp = len(flagged) - tp
    fn = sum(y for p, y in scored if p < t)
    precision = tp / (tp + fp) if flagged else 0
    recall = tp / (tp + fn)
    print(f"threshold {t:.1f}: precision {precision:.0%}  recall {recall:.0%}")

A lower threshold catches more defects (higher recall) and raises more false alarms (lower precision). The right answer is about cost, not maths. A missed defect that reaches a European buyer costs far more than a QC worker re-checking a false alarm, so the exporter sets the threshold low.

How they fail

Failure	Cause	Fix
Works in the lab, fails in the factory	Different lighting, camera, angle	Collect training data from the real site; fix the camera setup
OCR misreads Urdu or handwriting	Little training data for the script	Use models trained on Urdu; add human verification for key fields
Bias in face recognition	Under-representation of some skin tones or ages	Test accuracy per group; avoid high-stakes use without review
Privacy complaints	Cameras recording people without notice	Signage, data retention limits, blur faces where identity is not needed

Opening a bank account with a phone photo. A Karachi bank digitised its account opening. Customers photograph their CNIC with whatever phone they have. An OCR pipeline de-skews the image, reads the 13-digit number, name and date of birth, checks the number format, and compares the card photo with a selfie. Any field with confidence under 90% is shown back to the customer to confirm, and a person reviews the application if the face match is borderline. Onboarding fell from two days to under ten minutes, and the confirm step caught the worn, faded cards the OCR kept misreading. That step was the analyst's idea, not the vendor's.

Quick recap

• Classification, detection, segmentation and OCR are the core tasks, and a multimodal LLM can do all four for low-volume work.
• Capture conditions and pre-processing decide most of the real-world accuracy.
• Set confidence thresholds by the cost of a miss against the cost of a false alarm.
• Test under real conditions and per group, and handle privacy on purpose rather than by accident.

Homework

1. Run the threshold example and find the threshold that gives at least 80% recall with the highest precision.
2. Design a five-step pipeline for counting vehicles at a Rawalpindi toll plaza, noting one failure mode at each step.
3. A school wants face recognition for attendance. List two benefits, two risks and one alternative design.