> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?
I don't have social media outside of HN.
It comes from a few observations:
1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).
2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.
3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.
I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).