But this is still scientific communication. It's really nice that it's legible!
> Even though softmax1 is facially quite boring, I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research. If you want to run some experiments and prove me right, DM me on Twitter and we’ll get a paper going.
I'm guessing that in the stodgy world of science, a communication like this might happen over lunch at a conference, limited to a small clique of researchers who are zealously guarding their next paper. Who could blame them, publish or perish!
But someone will probably test this theory out (after my read, it will probably happen in llama.cpp with preliminary results on GPT-2 by next week) and achieve results, and it will happen quickly and legibly to the outside world, because this was published openly and without all of the pretension that formal science (tm) has. If it works, it works. Stuff like this is the soul of the internet. Sharing knowledge and making it legible for all.
Workshop submissions often don't need evidence. They just need a small kernel to spur discussion.
Without experiments, there is no hope of publishing this in anything more than a workshop. Nor should there be.
Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.
https://arxiv.org/pdf/2208.07339.pdf
Outliers appear at model size 6.7B and are not present at 2.7B