It sucks that we’re still at “best practices” phase. We’ve been in this phase for the last three decades [1], and I really hope we enter “good theory” phase soon.

[1] https://cseweb.ucsd.edu/classes/wi08/cse253/Handouts/lecun-9...

there's likely still an infinite amount of things to figure out, transformers haven't been out for 10 years yet

Why is that?