AI 101 — What is computer vision?

Computer vision is the field of AI that takes images and video and turns them into decisions: this is a face, that is a stop sign, this scan shows a fracture. The camera is not the hard part. The hard part is that a photo is just a grid of numbers, and software has to work out what those numbers mean.

Why it matters now

Vision is the AI most people already use without calling it AI. Your phone unlocks on a face, sorts photos by who is in them, and lets you search "beach" across 20,000 shots. A warehouse robot picks the right box because a camera told it which one. Radiologists in several countries read scans alongside software that flags suspected nodules.

The field is also where AI's failures are most visible, because a wrong answer is easy to see and hard to argue with. And it is where the money sits: image and video understanding is the part of AI that has to work before self-driving cars, delivery drones, factory automation or a useful AI glasses product can ship. Everything the industry says about "AI that understands the world" runs through it.

The mental model: pixels into labels

An image is a rectangle of pixels, each one three numbers for red, green and blue — a 12-megapixel phone photo is roughly 36 million numbers. A vision model's job is to shrink that flood of numbers down to something short: a label, a set of boxes, a mask, a count.

It does that with layers. The first layer looks at tiny patches — maybe 3×3 pixels at a time — and learns to respond to simple patterns like a vertical edge or a patch of colour. The next layer looks at the responses of the first, so it can spot corners and curves. Later layers combine those into textures, then parts, then whole objects. Nobody programs "an ear looks like this"; the pattern detectors emerge from training on millions of labelled examples.

The sliding window that produces this is called a convolution, and the model is a convolutional neural network. Since 2020 the same job has increasingly been done by vision transformers, which chop the image into patches and let every patch compare itself with every other — better at context, hungrier for data. Both approaches output the same kind of thing: a score for each possible answer. "Cat" at 0.94, "fox" at 0.03.

The analogy: an assembly line of inspectors

Picture a factory line of inspectors, each with one small stencil. The first inspector only checks for edges and passes a map of where it found them to the next station. The second looks at that map for corners, the third for round shapes, the fourth for fur texture, and so on down the line. The final inspector reads the accumulated notes and calls it: a cat, a dog, a cushion.

That is why vision models are fast and why they are literal. Each inspector is excellent at its one narrow check and knows nothing about the others. Show the line a cat wearing a hat in a snowstorm and the notes may add up to something strange, because no station ever saw a cat, only features.

The founding demonstration of how well this scales was AlexNet in 2012, which classified 1.3 million images into 1,000 categories with a top-5 error rate of 18.9%, far better than anything before it, according to the paper whose 60 million parameters were trained on graphics cards. That gap — a general pattern-learning machine beating hand-written rules — is the moment modern vision started.

Common misconceptions

"It sees what I see." It sees numbers. It has no idea that a cup holds coffee, and it will happily label a printed photo of a cat as a cat. Understanding what an object is for is a separate problem, which is why vision alone does not make a robot useful.

"More data fixes everything." More data of the same kind does not fix a model that has never seen your lighting, your camera angle or your factory floor. The standard failure is distribution shift: near-perfect on the test set, unreliable on the night shift. Medical imaging models are notorious for learning the scanner's brand as a shortcut instead of the disease.

"It can't be fooled by something invisible." Small, deliberately crafted changes to pixels — noise a person cannot see — can flip a confident answer, which OpenAI explained in a 2017 writeup on adversarial examples. A printed sticker has been shown to turn a stop sign into a speed-limit sign for a vision system. This is why safety-critical deployments pair cameras with radar, lidar or plain rules.

"Face recognition and face detection are the same." Detection finds a face in the frame. Recognition decides whose face it is by comparing a numeric fingerprint against a database. The second one is the privacy story — the matching, not the camera — and it is the mechanism behind deepfakes being detectable at all, since a generator's output leaves different fingerprints than a lens does.

Where to learn more

The AlexNet paper is short, readable and still the cleanest explanation of why layers of simple detectors beat clever hand-written rules. Stanford's CS231n course notes remain the best free walkthrough of how the pieces fit. And our own explainer on multimodal AI covers what happens when a vision model is wired to a language model — the setup behind almost every product that can now "look at" a picture and talk about it.

Should a vision model used in a hospital be required to show what it looked at before it answers? Write to us via our contact page.