You don't learn from being right

💫 Learning to climb one missed prediction at a time, training neural nets, postcard from Jakarta

When I’m climbing, there are two types of magical moments:

• when I think a move will work, but it doesn’t, and I fall 😯

• when I think I’m going to fall, but it works out, and I don’t 💫

Both update my mental model of what I can and cannot do.

Mistakes are interesting, because that’s how we learn.

Neural networks

The same is true for training neural networks.

When training, we run the neural net on a series of example data points. For each example, we calculate the prediction and compare it with what we know is the right answer.

• ✅ If the prediction is perfect, we don’t change anything.

• 🔧 If the prediction is not perfect, we adjust the network weights a little bit. The goal is to make the prediction (for that specific example) slightly more correct the next time.

For example, consider the word-predicting neural network we trained on the complete works of Shakespeare:

One example data point in the training set would be:

Input (three word triplet): “to”, “be”, “or”
Output (the following word to be predicted): “not”

If the network already predicts ‘not’ with full certainty for this triplet, we leave the weights unchanged. It’s only when the prediction is not correct that we adjust the parameters. And the more incorrect the prediction is, the bigger the adjustment.

Conclusion

The original neural network is the one in our brain. All our experiences, memories and competencies are stored in the strengths of connections between neurons. We “train” this network every day through our lived experience.

And as the common wisdom goes, we learn the most from mistakes 💫.

Postcard from Jakarta

I’m starting what was supposed to be a trip to climb Puncak Jaya; but through a last minute change of plans will be a more conventional trip around Indonesia 🏖. Starting in the nation’s bustling capital. At ~32 million, Jakarta is the 2nd biggest urban area in the world!

Keep learning 💫,
– Przemek