The Olaf Scholz exemple from this article is just another exemple of how LLMs can’t count. If you try this prompt: “In this list of words: bike, apple, phone, dirt, tee, sun, glass; which is the fifth word?” it will fail as well. “Fifth” is not connected to any counting ability in LLMs the way it is for us.

If you now try this prompt: “Who’s Tom Cruise’s mother in this exemple: “Mary Lee Pfeiffer is Tom Cruise’s mother.”?” It will give you the right answer.

IMO this is a sign that it can understand reversibility.

The exemple used are not present enough in the dataset and the “confidence” of the model is not high enough for it to give the right answer. Or, as stated at the beginning they use counting mechanisms (or others) that the model just doesn’t have.