I'm in no way an expert but I feel that today's LLMs lack some concepts well known in the research of logical reasoning. Something like: semantic.
And what's remarkable about LLMs is exactly that: they don't reason like machines. They don't use the kind of hard machine logic you see in an if-else chain. They reason using the same type of associative abstract thinking as humans do.
"[LLMs] reason using the same type of associative abstract thinking as humans do": do you have a reference for this bold statement?
I entered "associative abstract thinking llm" in a good old search engine. The results point to papers rather hinting that they're not so good at it (yet?), for example: https://articles.emp0.com/abstract-reasoning-in-llms/.
But the closest thing is probably Anthropic's famous interpretability papers:
https://transformer-circuits.pub/2024/scaling-monosemanticit...
https://transformer-circuits.pub/2025/attribution-graphs/bio...
In which Anthropic finds circuits in an LLM that correspond to high level abstracts an LLM can recognize and use, and traces down the way they can be connected. Which forms the foundation of associative abstract thinking.