It is definitely a nice milestone, for researchers.

I do think, though, that one of the rewards of making LLMs promising was the discovery that the bar for “what is useful” for any task is higher than what the average person can do. That could be seen as moving the goalposts, but another way of looking at it is: we’ve finally bothered to install a scoreboard.

Like, if a person can’t solve a quadratic equation, they probably just aren’t useful for manipulating equations. Which is fine, of course, the average person is also not a very good hammer or screwdriver.

We’re all specialized, the bar for usefulness is probably something more like “how would a person who’s taken an intro college course on this do it.”