Compare the models

Two models, drawn to scale

The program brnfck-qwen2.5-0.5b runs Qwen2.5-0.5B; brnfck-qwen3.5-0.8b runs Qwen3.5-0.8B. Both have 24 layers, and each layer has two parts: one that mixes in the earlier word pieces, and a feed-forward part that works on the word piece alone. Below, every weight of the two models takes the same room, so the higher a block, the more weights it holds. Within a layer, the width of each part is its share. Click a layer to see what is in it:

From Qwen2.5-0.5B to Qwen3.5-0.8B, three things changed. The table of word pieces, used at the start to look up the word piece that comes in and at the end to score every word piece that could come next, has almost twice as many weights: 248,320 word pieces of 1,024 numbers against 151,936 of 896. It is a third of Qwen3.5-0.8B. The feed-forward parts of Qwen3.5-0.8B are a little smaller. And the parts that mix in the past grew five times: 233.8 million weights in Qwen3.5-0.8B against 44.1 million in Qwen2.5-0.5B.

The six layers of Qwen3.5-0.8B that look back hold 44.0 million of those, as many as all 24 of Qwen2.5-0.5B. The other 189.8 million are in the 18 DeltaNet layers. A DeltaNet layer works out a key, a value and a question of 128 numbers for each of its 16 heads, and a gate for what comes out: 10.5 million weights. A layer of Qwen2.5-0.5B that looks back works out 14 questions of 64 numbers, but only 2 keys and 2 values: 1.8 million.

Why go to all that trouble?

Looking back gets heavier with every word piece

In SmolLM2-135M, Qwen2.5-0.5B and Qwen2.5-1.5B, every layer looks back at all earlier word pieces. For that, each word piece leaves numbers behind in every layer: a key, which says what it is about, and a value, which says what it brings. They stay on the tape until the answer is done. Every new word piece adds more of them, and every new word piece walks past all of them.

Qwen3.8-27B has 64 layers. 16 of them look back in this way. The other 48 are DeltaNet layers: they keep one memory of a fixed size and nothing per word piece. Move the slider to make the conversation longer:

In Qwen3.8-27B, a layer that looks back keeps 2,048 numbers for every word piece. A DeltaNet layer keeps 786,432 numbers from the first word piece on, and still 786,432 after a million. Up to 384 word pieces the looking back is the smaller one. After that the DeltaNet layer is, and the gap only grows. For the whole of Qwen3.8-27B: the 16 layers that look back keep 32,768 numbers per word piece, the 48 DeltaNet layers 37,748,736 numbers in all.

And in Qwen3.5-0.8B, the small model of brnfck-qwen3.5-0.8b?

In SmolLM2, Qwen2.5 and Qwen3.5

In Qwen3.5-0.8B the sums come out differently. Its 6 layers that look back keep 6,144 numbers for every word piece: 2 keys and 2 values of 256 numbers each. That is exactly what Qwen2.5-0.5B keeps, in 24 layers with 2 keys and 2 values of 64. On top of that come the 288 tables of 16,384 numbers, and a short window: before its table, a DeltaNet layer mixes every number with the same number of the three word pieces before it, so it keeps those three. That is 5,050,368 numbers from the first word piece on.

Below, the models of brnfck-smollm2-135m, brnfck-qwen2.5-0.5b, brnfck-qwen2.5-1.5b and brnfck-qwen3.5-0.8b side by side, on a scale where every line is a factor of ten. Move the slider to make the conversation longer, and switch to the work a word piece costs:

So in Qwen3.5-0.8B the fixed memory saves nothing against Qwen2.5-0.5B, at any length. It keeps less than Qwen2.5-1.5B from 617 word pieces on, and less than SmolLM2-135M from 940. The chat programs stop at 256 word pieces. The gain is in the large models: in Qwen3.8-27B a layer that looks back keeps more than a DeltaNet layer after 384 word pieces, as above.

The work tells the same story. Each weight is used once for every word piece: 752 million multiplications for Qwen3.5-0.8B, and 18.9 million more for its tables. Looking back costs 24,576 multiplications per earlier word piece, so at 256 word pieces it is less than 1% of the work. Only after about 31,000 word pieces would it cost as much as the weights. For Qwen2.5-0.5B that point comes at about 11,500.

A memory of a fixed size cannot keep everything. It has to choose what to keep and what to let go. How it does that is the rest of this page.

What is in those 16,384 numbers of a table?

A table instead of a list

A DeltaNet layer has several heads, and each head has a table of 128 by 128 numbers. Qwen3.8-27B has 48 heads in each DeltaNet layer: 48 tables of 16,384 numbers is the 786,432 above. The table is the whole memory of the head. Like a layer that looks back, the head works out from each word piece a key, a value and a question, each 128 numbers. Then it does four things:

1. Fade. Every number in the table is multiplied by the decay, a number between 0 and 1 that the head works out anew for each word piece. Near 1 it keeps almost everything, near 0 it wipes the table.
2. Ask. The table multiplied by the key gives what the table already holds under that key.
3. Write the difference. The value minus that answer is added to the table along the key, times the write strength, from 0 to 1. Writing the same thing twice changes nothing the second time. This difference is the “delta” of the name, and the fading is the “gated”: the model is a Gated DeltaNet.
4. Read. The table multiplied by the question gives what the head passes on.

Below are two real tables of Qwen3.5-0.8B, the smallest model with the same layers as Qwen3.8-27B. Each square is one number, orange above zero and blue below; I'm showing every fourth row and every fourth column, 32 by 32 of the 128 by 128. Press Play and watch the tables while Qwen3.5-0.8B reads the question and writes its answer:

The two heads were picked from the 288 of Qwen3.5-0.8B: of the heads that hold the most at the end, the one that forgets fastest and the one that forgets slowest. The half-life under each table is the number of word pieces after which half of what it holds has faded, if nothing new comes in. The left one keeps two thirds to four fifths of its table from one word piece to the next, so it holds mainly the last few. The right one, in the very first layer, fills up while the question is read and keeps that through the whole answer: its half-life is more than 100,000 word pieces.

No head chooses one or the other for good. The decay is worked out for every word piece. Head 0 of layer 17 has a half-life of 40 to 70 word pieces, until a sentence ends: at every full stop of these three answers it drops below one word piece, and the table loses about half of what it holds.

What happens inside one table at one word piece?

One word piece, step by step

Here is one head of Qwen3.5-0.8B at work, with all 16,384 numbers of its table. The four steps are numbered as in the list above. Each row of the table belongs to one number of the key, each column to one number of the value. Asking with a key adds up the rows, each times its number of the key; writing adds the key times the difference, row by row. Pick a head, click a word piece or press Play:

Watch head 8 of layer 0, which never fades. The first time the comes in, its value differs from what the table holds under its key by 69%. Later in the answer the difference is 3% to 27%: the table already holds most of it, and the write is small. Writing the same thing twice changes nothing. Head 0 of layer 17 does the opposite at the full stop after Paris: it keeps 28% of its table and writes into one that is nearly empty.

Two things around the table are also new in Qwen3.5. Before it, the key, value and question each pass the short window of the last four word pieces. After it, what the head reads is brought to one size and multiplied, number by number, by a gate that the layer works out from the word piece itself: silu(z), near 0 for numbers it shuts and close to z for numbers it lets through.

Those are three heads. What do the other 285 do?

All the tables at once

Below, each row is one layer of Qwen3.5-0.8B and each square one head. The colour says how much a table holds: the closer to the colour of the page, the emptier the table. (The size of a table is the square root of the sum of its 16,384 squares.) The thin rows are the layers that look back; they have no table. Press Play, and click a square to follow that head:

Most tables fill up within the first two word pieces. Layers 1 and 2 then flicker: many of their heads forget within a word piece or two. Layer 0 already holds on, and in layers 16 to 21 many heads stay full to the end. No head stays empty; the emptiest are in layers 1, 10, 13 and 14. Click head 0 of layer 17 and play: its table drops at every full stop. Within one layer, every head decides for itself what it keeps.

The colours change, but the number of squares does not. With a question ten times as long the map would have the same 288 squares, and Qwen3.8-27B would have 48 rows of 48.

One thing on this map is not part of Qwen3.5-0.8B. What are the yellow rings?

Fading in whole numbers

A brainfuck program has no fractions. Its cells hold whole numbers, and the programs keep a number such as 0.97 as a whole number with an agreed number of binary places. The recordings on this page come from the step before brnfck-qwen3.5-0.8b: the DeltaNet layers written out in whole numbers, with every product and sum in a cell of 32 bits, as the programs need. Then came brnfck-qwen3.5-0.8b itself, 7,709,034,855 characters, with a tape of 1,067,424,772 cells. It was checked: after its writing part the tape equals Qwen3.5-0.8B as loaded, in 1,018 sums, and it prints the reference answer. brnfck-qwen3.5-0.8b computes its own whole-number recipe, compiler/qwen3.5/engine/spec.js, bit for bit: whole numbers in every layer, not only in the DeltaNet layers, and weights of 10 bits. That recipe picks the same next word piece as Qwen3.5-0.8B in floating point at 214 of 225 positions of its answers (95.1%); with weights of 8 bits it would be 209.

The Chatbot loads brnfck-qwen3.5-0.8b again for every question, 2 min 10 s to 2 min 37 s each time. brnfck-smollm2-135m puts back a copy of its whole tape after an answer and goes on; brnfck-qwen2.5-0.5b and brnfck-qwen2.5-1.5b put back only the cells an answer changes, which their meta files list as restore ranges. The meta file of brnfck-qwen3.5-0.8b has no restore ranges yet. Ask brnfck-qwen3.5-0.8b in the Chatbot.

The fading is the hard part. Multiplying all 16,384 numbers of a table by the decay for every word piece costs as much as the reading and writing together. Worse, with a decay close to 1 a small whole number is rounded back to itself: 5 times 0.9999 is still 5 after rounding, for ever. Such a head would never forget.

So each head keeps one extra number, its scale. The table is the scale times a table of whole numbers. The fading multiplies only the scale. When the scale has fallen below one half, it is multiplied into the table once and set back to 1: the table is folded in. That is the yellow ring on the map. The chart beside the map draws the scale of the chosen head as a saw: down with each word piece, up at each fold.

A head that forgets fast folds at three word pieces in four. A head that keeps folds about once in forty, and one head in five never folds in these answers. On average a head folds once every 4.6 to 4.8 word pieces, so at each word piece the expensive multiplication is done for about a fifth of the tables.

And the one layer in four that still looks back: is it the same as in Qwen2.5?

The layers that still look back

Not quite. The six layers of Qwen3.5-0.8B that look back work as in Qwen2.5, but their heads changed in four ways:

1. Longer. A head of Qwen3.5-0.8B has 256 numbers instead of the 64 of Qwen2.5-0.5B, and a layer has 8 question heads instead of 14.
2. A norm instead of a bias. The question and the key of each head are brought to one size, and each number is then multiplied by its own gain. In Qwen2.5 a fixed number, the bias, is added to each of them instead.
3. Only a quarter turned. The position of a word piece turns only the first 64 numbers of a head, in 32 pairs. The other 192 carry no position at all.
4. A gate on the way out. What a head brings back is multiplied, number by number, by a gate between 0 and 1 that the layer works out from the word piece.

Below, one head of Qwen2.5-0.5B and one of Qwen3.5-0.8B, drawn to the same scale, then the gains and the gate of one head of layer 23 of Qwen3.5-0.8B, the last layer that looks back, while it reads the question and writes the answer:

The gate is mostly shut. In layer 23 it lets through 17% on average, in layer 3 only 8%. In every one of the six layers more than a third of all numbers get less than a tenth through, and some stay shut for the whole answer. Heads 2 and 6 of layer 23 are the most open: about a quarter of their numbers get more than half through. A head of Qwen2.5 has no gate: everything it reads goes on.

The turning: number f of the first 64 turns together with number f + 32, each pair at a speed of its own. The fastest pair turns one radian, about a sixth of a circle, per word piece; the slowest goes round once in 38 million word pieces. In Qwen2.5-0.5B the slowest pair goes round once in 4 million. The norm keeps the questions and keys of a head the same size at every word piece; before it, the questions of layer 23 range from 6.5 to 24.6.

Ask it yourself

Everything above was recorded. Here the model itself runs, Qwen3.5-0.8B, in this page, with the same code that made the recordings and a live look inside the DeltaNet memory. This is not brnfck-qwen3.5-0.8b. It is Qwen3.5-0.8B in floating point with its 18 DeltaNet layers in whole numbers, with 8-bit weights unless you choose otherwise: close to the program, but not what it computes. brnfck-qwen3.5-0.8b does every layer in whole numbers with 10-bit weights, and runs in the Chatbot.

This demo downloads the weights of the text part, 1.50 GB, and needs about 2 GB of memory; brnfck-qwen3.5-0.8b in the Chatbot downloads 1.36 GB and needs about 5 GB. Your browser does all the work: your question is not sent anywhere.

Choose how the 18 DeltaNet layers count: in floating point, as the model was trained, or in whole numbers, as a brainfuck program would, with 8 or with 10 binary digits for each weight. The layers that look back and the rest of the model stay in floating point. Then load Qwen3.5-0.8B, type a question and press Ask:

The answers differ a little between the three ways of counting. With whole numbers and 8-bit weights, 97 of every 100 word pieces are the same as in floating point; with 10-bit weights the first answers match word for word up to the last word or two. That is the error the whole-number layers bring. brnfck-qwen3.5-0.8b goes further, with every layer in whole numbers and 10-bit weights: its recipe picks the same next word piece as Qwen3.5-0.8B in floating point at 95.1% of the positions tested (214 of 225).

Every program at one scale

This page compared the models that the chat programs run. The site has fourteen brainfuck programs in all: six that run a chat model, trained on graphics cards elsewhere; four that train a network themselves, from random numbers, on a list of names or on Shakespeare; and their four children, the programs those trainings print, which write with what they learned. Below, all of them side by side, on a scale where every line is a factor of ten. A child is shown in its best version.

Qwen2.5-1.5B has two million times as many weights as the network that brnfck-train-names-bigram-756 trains, and brnfck-qwen2.5-1.5b is nine thousand times as long as brnfck-train-names-bigram-756. Most of a chat program writes its weights onto the tape; a training program is nearly all engine, because its weights start as random numbers it draws itself. What it does with them, the chat programs never do: for every letter it reads, it goes back through its own network and changes every weight a little.

Where to go next

The Chatbot runs the brainfuck programs themselves. How it works follows one question through a program, Inside the program takes one apart, and the map shows one whole.

1. ,Ask brnfck-qwen3.5-0.8bChatbotRuns Qwen3.5-0.8B, the model of this page, in your browser: about 5 GB of memory, ready in 2 min 10 s to 2 min 37 s.
2. -How it worksArticleOne question followed from your keyboard to the printed answer. No programming needed.
3. -Inside the programArticleTake the writing part and the engine apart, with figures drawn from the program itself.
4. <>Program mapThe whole programSee every character of a program in one picture, and zoom in until you can read the commands.
5. +LearnTwo coursesStart from the eight commands, then build the copy, the multiplication and the search for the highest score.

Notes and sources

1. Qwen3.5-0.8B, the model recorded here and run by brnfck-qwen3.5-0.8b, and Qwen3.8-27B, a much larger model with the same kind of layers. Made by the Qwen team of Alibaba Cloud. Qwen3.5-0.8B is published under the Apache 2.0 licence: huggingface.co/Qwen/Qwen3.5-0.8B. The numbers of Qwen3.8-27B (64 layers, 48 DeltaNet layers with 48 value heads of 128, 16 layers that look back with 4 key and value heads of 256) are from its configuration file.
2. Gated Delta Networks: Improving Mamba2 with Delta Rule, Songlin Yang, Jan Kautz and Ali Hatamizadeh, 2024. The layer as Qwen3.5 uses it.
3. Linear Transformers Are Secretly Fast Weight Programmers, Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, 2021. The write-the-difference rule in this form.
4. The recordings. compiler/qwen3.5/compare/record.js runs Qwen3.5-0.8B with its 18 DeltaNet layers in whole numbers (compiler/qwen3.5/reference/deltanet.js, 8-bit weights, 32-bit cells) and greedy choice of the next word piece. For every word piece it stores the size of all 288 tables, their decay, write strength and scale, and every fourth row and column of two tables. The folds are counted again in the recorder and compared with the count of the layer itself, word piece by word piece. compiler/qwen3.5/REPORT.md compares the whole-number layers with the model in floating point.
5. The shapes and the steps. compiler/qwen3.5/compare/shapes.js counts the weights of SmolLM2-135M, Qwen2.5-0.5B, Qwen2.5-1.5B and Qwen3.5-0.8B from the headers of their weight files, and works out from their config.json what each keeps and how many multiplications a word piece costs. compiler/qwen3.5/compare/record-parts.js runs Qwen3.5-0.8B in floating point on What is the capital of France? and stores, for three DeltaNet heads, everything that goes into their tables, and for the six layers that look back the size of their questions and keys and, for layer 23, the gate of every number.