Google is large enough, well-funded enough, and the opportunity is great enough to run experiments.
You don't necessarily have to prove it out on large foundation models first. Can it beat out a 32b parameter model, for example?
You don't necessarily have to prove it out on large foundation models first. Can it beat out a 32b parameter model, for example?
While they do have lots of money and many people, they don't have infinite money and specifically only have so much hot infrastructure to spread around. You'd expect they have to gradually build up the case that a large scale experiment is likely enough to yield a big enough advantage over what's already claiming those resources.
> In fact, the UL2 20B model (at Google) was trained by leaving the job running accidentally for a month.
https://www.yitay.net/blog/training-great-llms-entirely-from...