LearnBuild toward a chatbotLooking back
Looking back
Aim. At the end of this lesson your program looks back at four earlier word pieces, decides how much each one counts, and mixes what they have to say into one number.
Where the lessons are going
The sum so far was given its four inputs. In a model they come from the text before: here the words of "The capital of France is", cut into word pieces, which are words or parts of words. To go on, the model must look back at those pieces.
Decide which piece counts
Try it: four earlier word pieces have a score each, and each says one number. Move the scores until " France" is the only piece that counts: its count 15, the other three 0:
| What the pieces say, mixed by count | |
| The same, from the brainfuck program |
What you just saw
Every earlier piece has a score: how useful is it right now? And it has something to say, here one number.
The piece with the best score always gets the full count, 15. Every other piece gets less, and only the distance to the best matters. A piece far behind gets 0 and is not heard at all. The result is the average of what the pieces say, each counted as often as its count.
This looking back is called attention.
A list of ready-made answers
Try it: a piece that is far enough behind the best gets count 0. Find the smallest distance at which that happens:
What you just saw
In the original model, the count comes from a curve that needs numbers with fractions. Here it comes from a list of 64 ready-made answers. The distance is divided by 8, and that picks an entry of the list. Such a list is called a lookup table.
You found its edge. At 160 points behind, a piece is cut off. One point closer and it still gets count 1.
Looking up is cheap because of the peel (step 3 of lesson 5): the bits of the entry number are read one by one, and each bit halves the list.
Watch the program walk past the pieces
Try it: choose a fast speed and press Play, or go to the last step. The pointer passes the four pieces twice:
It uses the scores you set in step 1. The program is characters long, so only the part around the lit-up command is shown.
What you just saw
Everything in this program is something you have built.
- First pass: find the best score. A "best so far" travels from piece to piece (step 3 of lesson 7). At each piece the program works out how far this score is above it, clamps that at zero (step 5 of lesson 5) and adds it.
- Second pass: the counts. Best minus my score, divided by 8 (a peel), then the lookup table of step 2.
- Count × what the piece says is a product of two numbers from the tape (step 4 of lesson 6). The total walks along.
- The average. There is no division by an unknown number. A second small table gives 4,096 divided by the total count. Multiply by that, then throw away 12 bits.
One copy of the program handled all four pieces, because it sits in a loop that walks from frame to frame (step 2 of lesson 7).
What you can do now
Look back along the tape, weigh the earlier word pieces and mix them. The numbers that go into the sum now come from the text itself.
How the real program differs
The cost of a lookup. In the chat program, looking up one of 256 values takes about 68 steps with the peel (step 3 of lesson 5), against about 500 by counting down.
Where the scores come from. Here you set them. In the chat program each score is itself a sum of products between the current word piece and an earlier one. The bits of the current piece are peeled once and used for all the earlier pieces (the note "A bonus" of lesson 6).
A difference with the original. The original model never cuts a piece off completely; it lets tiny counts through. This is one of the places where the whole-number model differs a little. The chat model uses a finer table than this lesson: 256 entries, and the best piece gets count 255.
What a piece "says" is in reality a whole row of numbers, not one. The same counts are used for every number of the row.