LearnBuild toward a chatbotOne number of the model
One number of the model
Aim. At the end of this lesson one program does the whole sum, from four inputs to the number 93. Then it does three such sums at once.
Where the lessons are going
The four obstacles (lessons 4 to 8) are behind you, and every part exists: multiply and add (step 4 of lesson 3), divide, round and clamp (step 5 of lesson 5). So far they ran one at a time. Now they run as one program.
The whole sum in one go
Try it: press Play, or go to the last step. Four inputs, 90, −25, 40 and 12, go in. Does the result cell end at 93?
Inputs on the tape: 90, −25, 40 and 12. Weights, in the program: 100, −60, 30 and 14.
What you just saw
Three things, one after the other. First the four products were added into one total: 11,868. Then the total was moved to the top of a cell and its bits were peeled off: that divided it by 128 and rounded it. Last, the program checked that the result fits between −128 and 127.
One such sum is called a neuron. A model is this, done a very large number of times. Each result becomes an input for later sums.
What whole numbers cost
Try it: move the sliders until the result is exactly 0, while no input is 0, no weight is 0, and the sum is not 0 either:
| Input | Weight | Input × weight |
|---|
| The sum | |
| Divided by 128 and rounded | |
| Result of the brainfuck program |
What you just saw
A sum that is less than half of 128 rounds to 0. Whatever it had to say is gone. The clamp of step 5 of lesson 5 throws something away too: every sum above the ceiling gives the same 127.
That is the price of working with whole numbers only. The original model uses numbers with fractions. Brainfuck has none, so the model was first rewritten in whole numbers. That version picks the same next word piece as the original about 95 times in 100.
Three numbers at once
Try it: press Step once and look at input 1 and the three totals. Then go to the last step:
The same four inputs, and three rows of weights: one row each for the word pieces " Paris", " London" and " cheese".
What you just saw
The three sums were not made one after the other. Each input was emptied once, into all three totals at the same time, each with its own weight. That is the pass that feeds several cells, from step 3 of lesson 2. So the number of sums hardly matters: two steps for each input.
A set of sums over the same inputs is called a layer. This model has 30 layers, and the results of one are the inputs of the next.
Keep the three results in mind: 93, 41 and −49. In step 2 of lesson 11 they are the scores of three word pieces.
What you can do now
Work out a number of the model from start to finish, and many of them in one pass. The first two parts of the sum run as one program: 11,868, then 93.
How the real program differs
Rounding. Adding half of the divisor first (64 for 128) turns "round down" into "round to the nearest".
Numbers below zero need nothing special in the sum. On a cell that goes round, adding "the top minus 3" is the same as taking 3 off.
Where the weights are. In this lesson the weights stand in the program, as in lesson 3. In the chat program they are numbers on the tape (step 3 of lesson 8), and it makes the same sums by walking along rows of weights.