LearnBuild toward a chatbotObstacle 4: size
Obstacle 4: size
Aim. At the end of this lesson you know where 135 million weights go, why the program is a billion characters long and not 43 trillion, and how the program finds the end of a row of weights.
Where the lessons are going
Every piece of arithmetic works, anywhere on the tape. What is missing is the model itself: a very long list of numbers, the weights. In lesson 3 a weight was a row of plus signs in the program.
Eight weights, written by counting
Try it: go to the last step. The program writes eight weights onto the tape. Then look at how long it is:
This program is characters long: for each weight.
What you just saw
A number costs as many plus signs as it is large. That was step 3 of lesson 1. Eight weights already take more than a thousand characters.
The chat model has 135 million weights, and the tape needs more numbers besides. Written by counting, the chat program would be about 43 trillion characters long.
The same weights, written with digits
Try it: press Play. Each cell gets a small number. Then one loop multiplies the whole row by five. Watch that happen three times:
| Characters of program | By counting | With digits |
|---|---|---|
| These 8 weights | ||
| A row of 8,192 weights |
What you just saw
We write 93 with two digits, not with ninety-three tally marks: 9 tens and 3. The program does the same with fives: a small digit, times five, the next digit, times five, the last digit.
"Times five" for the whole row is one small loop: [[−<+++++>]>]. It is written once, however long the row is. Each pass also moves the row one cell to the left, which is why the row starts three cells too far to the right.
The chat program writes its tape like this, in rows of up to 8,192 cells, with about six characters for each number. That is why it is 1 billion characters long and not 43 trillion.
Walk along the weights
Try it: press Play. Four weights lie on the tape and the input is 5. Watch the pointer walk along the weights and back, twice:
The weights are 100, −60, 30 and 14. To keep the example short they come in as typed input.
What you just saw
The weights are on the tape now, not in the program. So weight × input is a product of two numbers from the tape (lesson 6). The input 5 has two bits on: 4 and 1. For each of them the pointer walked along the row once and added "weight × that worth" to every result.
Look for the walk in the program: [ … >]. It says: as long as this cell is not zero, do the work and step right. The program does not count the weights. It walks until it meets an empty cell.
This program never changes when the weights change. Such a part is called the engine. It runs any weights, the way a music player plays any song.
Break the walk
Try it: every weight takes a number from −128 to 127. One of those numbers makes the engine go wrong. Find it:
Hint
The walk stops at an empty cell. Look at the column "On the tape". Which weight would be put on the tape as 0?
What you just saw
A weight can be zero or below zero. But an empty cell means "end of the row". So every weight is put on the tape with 128 added. Then −127 to 127 become 1 to 255, and no cell of the row is ever empty.
You found the one number that lands on 0. The walk stopped there, the pointer was not where the program expected it, and everything after that went wrong. The model's weights simply never have that value.
The added 128 makes every result too large. So at the end the program takes 128 × the input off every result, with one multiplying loop. That is also why the eight weights of steps 1 and 2 were all above zero: they were already written with the 128 added.
What you can do now
Put a long row of weights on the tape for a few characters each, and walk along it without knowing how long it is. The whole sum now works with weights from the tape. The fourth obstacle is gone.
How the real program differs
Step 2 is simplified. The real writer also uses digits below zero and other steps than five, and picks them for every row so that the text is as short as it can be. It reaches about six characters for each number. The loop that multiplies the row tests the cell it stands on, so no cell may be empty during the passes. Every cell therefore starts at 1 before the first digit is added.
Two parts. The chat program is 1,030,902,380 characters. The first 1,004,387,443 only write numbers onto the tape and read nothing. The engine after that is 26.5 million characters. Drawn to scale:
- writes numbers onto the tape: 1,004,387,443
- the engine: 26.5 million
Nearly all of the program only writes numbers. An earlier version had only the engine and read its weights as typed input, as step 3 does here: 355,761,802 bytes before your question.
The real walk. In the engine a walk costs two steps for each weight, and the row is not carried back each time: it goes to and fro between two places. The weights of the chat model are a little wider than those of this example, so they are put on the tape with 512 added, not 128.
What the first billion characters look like from the outside is shown in "A billion characters of numbers" in How it works.