> Except this LLM would have a real shot of beating humans in open-domain problem solving.

At some point we need to start recognizing LLMs for what they are and stop making outlandish claims like this. A moment of reflection ought to reveal that “open domain problem solving” is not what an LLM does.

An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data), in the absence of a prior formulation of these laws in the training set.

TFA describes a need for scoring, at scale, qualitative results to human queries. Certainly that’s important (it’s what Google is built upon), but we don’t need to make outlandish claims about LLM capabilities to achieve it.

Or maybe we do if our next round of funding depends upon it.