763 karma · joined December 7, 2020
Try putting on a dating website that you work at Google vs you work in agriculture and tell us which yielded more dates.
Sidenote: this website is unreadable on a mobile browser due to popups, auto-playing videos, etc.
That's not the main point even though it probably helps. As OkayPhysicist said above, without a nonlinearity, you could collapse all the weight matrices into a single matrix. If you have 2 layers (same size, for simplicity) described by weight matrices A and B, you could multiply them and get C, which you could use for inference.
Now, you can do this same trick not only with 2 layers but 100 million, all collapsing into a single matrix after multiplication. If the nonlinearities weren't there, the effective information content of the whole NN would collapse into that of a single-layer NN.
"only" is doing a lot work here because that non-linearity is enough to vastly expand the landscape of functions that an NN can approximate. If the NN was linear, you could greatly simplify the computational needs of the whole thing (as was implied by another commenter above) but you'd also not get a GPT out of it.
I know I'm luckier than most humans on Earth, but still hedonistic adaptation is a thing, even if you grew up in a poor family.
I think he was generous enough with his time for creating the library in the first place and posting this update. It's not like he's owing anything to anybody. I don't understand how you can demand him to do further UNPAID work to recruit maintainers who would also do further unpaid work.