As an example, I clicked through to "english russian proverbs" which gives the model an english input and four russian proverbs, and asks the model to pick the closest one in meaning. There are 80 of these questions. My question is "why this task? why 80? why 4 options?" The authors give some nonspecific reasoning in the github page, but it really feels like people are out there creating random tasks as they think of them, running them all against models, and the ones that do better get added to a list to make headlines like "137 emergent abilities of language models", when the number of abilities models can do seems dominated by the number of "abilities" that have been invented, rather than any property of the model. I suspect that if we continued to use the same process to create twice as many "abilities", we would see something like twice as many that models can do.
The fact that the models get better at these constructed tasks when they have more parameters also seems like a no-brainer. You are adding coefficients to a function. A function with 1 coefficient will not be able to describe much. A function with 2 describes more. If you are memorizing the internet, having more coefficients means you can memorize more stuff, and even stuff with only a few examples can be better-fit in the billions of parameters. With growing training set size, and companies hiding their data, we also run into the issue of nobody really knowing what is in the training set. Did someone clone the big-bench repository and it got sucked into a training set and that's why the model does really well on its tasks after a certain number of parameters?