Looking at a picture

Squares of pixels

The program reads a picture of 256 by 256 pixels as 196,608 bytes: 256 squares of 16 by 16 pixels, every square its red pixels row by row, then its green ones, then its blue ones. Every byte is a number from 0 to 255, read with the comma command, one byte at a time, as the letters of a question are. The program has three commas: this one, and two that read the question after it, one for its first letter and one for the others.

The squares come in blocks of two by two: the four squares of a block follow each other. The model was made to take them in that order, because at the end every block becomes one word piece. Press Show the order below to see the program read the cat:

Every square looks at every square

In the text model a word piece looks back: it takes in what it needs from the word pieces before it, never from those after it. A picture has no before and after. In each of the 12 layers every square looks at all 256 squares, itself included, and takes in a share of each, the shares adding up to one. Pick a square in the picture above, and a layer with the slider: the brighter a square, the more of it the square you picked takes in. The figure opens on the eye on the left of the cat in layer 4: after itself, it takes in most from the other eye. In the last layer the other eye is what it takes in most of all.

In layer 2 more than half of what a square takes in comes from itself and the eight squares around it, in all four photos. By layer 10 that is a tenth or less: the squares look across the whole picture, on average eight or nine squares away. And from layer 3 on, the model uses a few squares as a shared notepad for the whole picture: every square takes in from them, whatever they show. In all four photos these are the squares in rows 2 and 3 of the right edge, in the later layers also row 11 of the left edge, and in the last layer the corners: background, whose own content the model can spare. By layer 10 the average square takes in 43% to 68% of everything from two or three of them. Larger models of this kind do the same; the paper that found it calls such squares registers. [4] In the figure they are ringed in blue and set apart, so that the rest shows where a square looks in the picture itself.

A program that runs one step at a time cannot look both ways at once, so it does it in two rounds per layer. In the first round every square works out its key and its value and leaves them in a slot of its own, 256 slots in a row, with its question and the numbers it carries. In the second round the squares come back one by one, take their question from their slot, and walk past all 256 slots: by then every slot is full. The looking itself is the same code as in the text models, which stops at the first empty slot. Here no slot is empty.

And after the last layer?

Four squares, one word piece

After the 12th layer every block of four squares becomes one row of 1,024 numbers: the four rows of 768 numbers side by side, through two more matrices. That gives 64 rows, as wide as the rows of the table of word pieces of the text model, and they go into a queue at the end of the tape.

The prompt has a place for them: 64 times the special word piece <|image_pad|> after <|vision_start|>, before the question. At each of those places the program takes the next row from the queue instead of the row of the table. From there on it is the text model of brnfck-qwen3.5-0.8b, the same 24 layers, with one change: its six layers that look back turn their keys by where a word piece is, and a piece of the picture is somewhere in two directions. Each one gets its row and its column in the grid of 8 by 8 blocks, as the model learned it.

What it said

Four photos, each with the question Describe this picture in one sentence., and two more questions: what colour the eyes of the cat are, and, in Dutch, where the photo of the windmills was taken. These are the answers the program printed.

The model is small and its answers can be wrong. Nobody sits on the bicycle. There is an animal lying in the meadow in front of the windmills; the model called it a cow when it ran in whole numbers, and left it out when it ran in floating point. Both are the model, not the brainfuck: the program prints the answers of its whole-number recipe exactly, and that recipe differs a little from the model in floating point, as the next section shows.

How it was checked

1. The model itself. The vision part was first written in JavaScript in floating point, from the code that Hugging Face publishes for the model, and checked against that code on the four photos: the squares in the same order, the same 85 word pieces of the prompt, rows within 0.01% of those of the original, and word for word the same answers.
2. The whole-number recipe. Brainfuck has whole numbers only, so the program computes a recipe in whole numbers (compiler/qwen3.5/vision/spec.js). Every LayerNorm became the norm of the text engines, after the mean was taken out of everything added to the stream, and every layer has its own widths, because the numbers grow from about 9 in the first layers to 2,240 in the last. On the four photos the whole-number model picks the same next word piece as the model in floating point 79 times out of 81 (97.5%); the text part of brnfck-qwen3.5-0.8b does so in 95.1% of its tested cases.
3. The brainfuck. On small models made up for the test, the brainfuck of the vision part gave exactly the numbers of the recipe, stage by stage, and left every scratch cell empty: 60 checks, the last of them with all 256 squares. With a small text model behind it, the whole program printed the recipe's answer: 3 more. That last test of 256 squares came after the first run of the real program, which left the tape once it had read the picture: it counted to 256 with the help of a cell that was in use, and the tests before it had at most 64 squares, a count that needs no help.
4. The program itself was checked as the other programs are: only the eight commands, the tape after its writing part equal to the tape of the loaded model, and every answer on this page equal, character for character, to the recipe's. compiler/qwen3.5/vision/verify_pure.js does all three.

What it costs

Where to go next

1. ,Ask brnfck-qwen3.5-0.8b-visionChatbotThe program of this page, in your browser: a desktop computer with about 7 GB of free memory, and patience.
2. []Compare the modelsArticleThe text part of Qwen3.5-0.8B: a memory of fixed size in three of every four layers.
3. -How it worksArticleOne question followed from your keyboard to the printed answer. No programming needed.

Notes and sources

1. Qwen3.5-0.8B, made by the Qwen team of Alibaba Cloud, published under the Apache 2.0 licence: huggingface.co/Qwen/Qwen3.5-0.8B. Its weight file holds the text model (752 million weights), the vision part (101 million) and a head that guesses two word pieces ahead (20 million), which this program does not run.
2. The photos, all from Wikimedia Commons and released into the public domain (CC0): Tabby cat with blue eyes by AdinaVoicu (Pixabay), The Windmills of Zaanse Schans, Netherlands by Mustang Joe, Bicycle 02 by Shams948, Apples fruit by safaritravelplus.
3. The code. compiler/qwen3.5/reference/vision.js is the vision part in floating point and compiler/qwen3.5/vision/hf_check.py the comparison with the Hugging Face code; compiler/qwen3.5/vision/spec.js is the whole-number recipe, gen.js its brainfuck, program.js the text engine with it, and compiler/qwen3.5/REPORT.md has the numbers.
4. Vision Transformers Need Registers, Timothée Darcet, Maxime Oquab, Julien Mairal and Piotr Bojanowski, 2023. Squares of background that the model uses for the whole picture, and what that does to its attention. The shares in the text above are worked out from the data of the figure.