Once again it's a kind of backhanded milestone for AI that in less than a year the goalposts have moved from "can it even once hold a half-coherent conversation" to "if anyone can think of a question it can't answer then it's dumb and not AI".
I do think, though, that one of the rewards of making LLMs promising was the discovery that the bar for “what is useful” for any task is higher than what the average person can do. That could be seen as moving the goalposts, but another way of looking at it is: we’ve finally bothered to install a scoreboard.
Like, if a person can’t solve a quadratic equation, they probably just aren’t useful for manipulating equations. Which is fine, of course, the average person is also not a very good hammer or screwdriver.
We’re all specialized, the bar for usefulness is probably something more like “how would a person who’s taken an intro college course on this do it.”
Has the Singularity already happened, and was it us?