How AI “Sees” an Image: Why It Chops Photos Into Visual Tokens
An AI image model does not necessarily begin by viewing a photograph as one complete scene. A modern vision system may first divide it into small square patches.
Those patches are converted into numbers, given information about their positions, and processed as a sequence of visual tokens. Here is how a photograph becomes something a transformer can work with.
This article begins a five-part series about how AI systems process images, represent visual ideas, make mistakes, and reveal what influenced their outputs.
When you look at a photograph, you probably experience it as one complete scene.
You may instantly notice a person, a cat, a tree, a road, and the relationship between them. You do not consciously inspect every pixel or divide the picture into a numbered grid first.
Many modern AI vision systems begin very differently.
A Vision Transformer can divide an image into small square sections called patches. It then converts each patch into numbers and treats the resulting collection as a sequence of visual tokens.
The image has not become language. The patches are not literal words. But they give the transformer manageable units that play a role loosely similar to the tokens used in text processing.
The model does not start with a complete scene
Imagine a photograph of a cat sitting beside a window.
You see the whole composition at once: the cat, the glass, the daylight, the curtain, and perhaps the fact that the cat appears to be looking outside.
A vision model receives pixel values. Each pixel contains numerical information about color and brightness. A large photograph may contain millions of these values.
Processing every pixel as a separate item would create an extremely long and expensive input. A patch-based system reduces that problem by grouping nearby pixels together.
For example, the model might divide the image into a grid of squares. Each square could contain 16 by 16 pixels, although the actual patch size depends on the model.
Picture a photograph printed on a chessboard. The AI separates the board into tiles, converts every tile into numbers, and then processes the tiles as a connected sequence.
Step 1: The image is divided into patches
Suppose an image is 224 pixels wide and 224 pixels tall. If it is divided into patches measuring 16 by 16 pixels, the result is a grid containing 14 patches across and 14 patches down.
That creates 196 patches in total.
Each patch contains a tiny section of the original photograph. One patch might include part of the cat’s ear. Another might contain a piece of the window frame. Another might show only a plain section of wall.
A single patch usually does not tell the model what the full image means. Meaning has to emerge from patterns and relationships across many patches.
Step 2: Each patch becomes a compact numerical representation
The pixels inside a patch are converted into a long list of numbers. The model then transforms that list into a more compact internal representation.
This representation is often called a patch embedding.
The important idea is simpler than the terminology: the model turns each image tile into a numerical package that the transformer can process.
It does not store a label such as “cat ear” inside the patch. The representation contains learned visual features that may become useful when combined with information from other patches.
This is similar to how a text model does not receive words as human-readable dictionary entries. It receives numerical representations that carry learned relationships.
You can read more about that broader idea in What Are Tokens? How AI Breaks Text Into Smaller Pieces.
Step 3: The grid is turned into a sequence
A transformer expects an ordered collection of tokens.
The two-dimensional grid of image patches is therefore arranged as a sequence. You can picture the rows of the grid being unrolled into one long strip:
Image grid:
Patch 1 | Patch 2 | Patch 3
Patch 4 | Patch 5 | Patch 6
Transformer sequence:
Patch 1 → Patch 2 → Patch 3 → Patch 4 → Patch 5 → Patch 6
This does not mean the transformer slowly examines one patch at a time in the same way a person reads each word aloud. Transformers can process relationships across the sequence together.
The sequence format simply gives the architecture an organized set of visual units to compare.
How does the model know where each patch belongs?
Turning a grid into a sequence creates an important problem.
The model must still know that one patch came from the upper-left corner while another came from the center or lower-right edge.
Without location information, the same collection of patches could potentially describe several different arrangements.
Vision Transformers therefore add positional information to the patch representations. This gives the system clues about where each patch appeared in the original image.
The same general problem appears in language. A transformer needs to distinguish “the dog chased the cat” from “the cat chased the dog,” even though both sentences contain similar words.
The article What Is Positional Encoding in AI? explains how models preserve information about order and position.
The transformer compares patches with other patches
Once the visual tokens and their position information enter the transformer, the system begins building relationships among them.
A patch containing part of an eye may become more meaningful when considered alongside patches containing fur, ears, whiskers, and the outline of a face.
A patch showing a round red shape may mean one thing beside a tree branch and something else beside a traffic pole.
The transformer’s attention mechanism helps different parts of the image influence one another. This allows the model to combine local details into broader patterns.
Attention does not mean human concentration or awareness. It is a mathematical process that changes how strongly token representations affect one another.
For a plain-English explanation, see What Is Attention in AI?.
Why patch size matters
Larger patches create fewer tokens. That can make processing faster and less expensive, but each token covers a larger section of the picture.
Smaller patches preserve finer detail, but they create a longer sequence that requires more computation.
This creates a practical trade-off:
- Larger patches: fewer tokens and lower processing cost, but less fine-grained detail.
- Smaller patches: more detailed visual input, but more tokens and higher processing cost.
The best choice depends on the task. Detecting a large object in a simple image may not require the same level of detail as reading tiny text or examining a medical scan.
Not every image model works this way
Patch-based Vision Transformers are important, but they are not the only form of computer vision.
Earlier and still widely used systems called convolutional neural networks process images using learned filters that move across local regions. Some newer models combine convolutional methods with transformer components.
Other multimodal systems may resize images, use patches of different sizes, or process visual information through several stages.
So it would be inaccurate to say that every AI model always chops every picture into identical squares.
The broader point is that an AI model needs to translate raw pixels into numerical units that its architecture can process.
Why this is not the same as human sight
Humans also process visual information through many stages. Our brains do not receive a ready-made label saying “cat.”
But human vision is connected to a body, attention, memory, movement, depth perception, and years of physical experience in the world.
A patch-based model starts from numerical image data and learned statistical relationships. It can become highly capable at classification, description, search, and visual question answering without seeing the world in the way a person does.
A Vision Transformer does not need to treat a photograph as one indivisible canvas. It can divide the picture into patches, turn those patches into visual tokens, preserve their positions, and learn relationships across the sequence.
The picture is reconstructed as relationships
At the beginning of the process, the system has patches and numbers.
As the information passes through the model, the representation can increasingly reflect edges, textures, shapes, objects, and relationships among parts of the scene.
The system does not literally rebuild the photograph like a jigsaw puzzle on a table. Instead, it builds a richer internal representation from interactions among the visual tokens.
This explains how an architecture developed for sequences can also work with images.
Researchers did not make the transformer grow human eyes. They translated an image into a structure the transformer already knew how to process.
Next in the series: Recognizing patches, shapes, and objects still does not guarantee that a system understands what a complete image means. The next article examines why AI can identify every object in a visual joke and still miss the point.
Comments
Post a Comment