Seems reasonable, and magical indeed. Can any expert on the topic comment on this hypothesis?
Seems reasonable, and magical indeed. Can any expert on the topic comment on this hypothesis?
I've worked in AI&ML since the 90s and both of these do seem much more intuitively promising than simply making larger nets. This seems borne out by the discussion in his previous post, that in Oct '21 OpenAI ran GPT-3 on it, solving 35% and they estimated:
"it appears likely that the 175 [billion link neural network] model would require at least two additional orders of magnitude of training data to reach an 80% solve rate.”
But instead 85% was achieved within a year using chain-of-thought, not a larger model.[2]
[1]https://twitter.com/PMayrgundter/status/1603224294124920832 [2]https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...