Grokking: A 777-Parameter Transformer Learned Real Math

> ...addition with 99.69% accuracy.

> Neural networks can learn algorithms, not just patterns. The 777-parameter model didn’t memorize. It couldn’t. It discovered a general procedure for addition that works on numbers it had never seen.

If that's true, I would have expected accuracy to be 100%. Why isn't it 100% accurate?

Are neural networks ever 100% accurate?

Executing an algorithm that you've learned and that is true should have 100% accuracy. Otherwise, I think you either haven't actually learned the algorithm or you learned an incorrect algorithm.

Ah, I see. As for my personal neural network: I learned long division in school and got the best possible grade in maths, and I still don't think I had 99.63% accuracy on long division.

Addition is way easier than division. :-)

How few parameter are necesary for binary adition? How does it escale with the base?

I'm glad it's not 737-Max-Parameter