In particular, the line about "what models can't do tells us what they don't know" is infuriating.
No, that's not the case at all. At least in a number of instances, what they can't do is because of what they do know.
As an example, one of thecan'ts I got from HN a year ago for GPT-4 was a variation of a classic logic puzzle. And indeed, the model can't solve it - nor can most major models since.
But it's not because the model can't solve the logic - it's because the token similarity to the standard form biases the output towards the standard solution. A hack as simple as changing the nouns to emojis can allow the model to get the correct answer and work through the logic successfully every attempt because it breaks that similarity bias.
People are way too confident around a topic where what's 'known' is more mercurial than maybe any field since 1930s particle physics.
I'd strongly recommend deleting 'never' or 'can't' from one's vocabularies on the subject unless one enjoys ending up with egg on their faces.