Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.
Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama.
Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.
No one really knows the answer to this question.
>Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained.
You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space.
It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum