>Maybe I need to reread his book, but I think of System 1 as "fast, reflexive, instinctive" mental activity, and System 2 as "slow, deliberative, rational, verbal" thought.

Current LLMs are the opposite of slow and deliberative, though? With the current transformer architecture, it seems like its function is more or less "look at this text and make a guess which word comes next", which is very much a fast and intuitive task. Any reasoning comes from CoT, which is more analogous to System 2 thinking. But that System 2 thinking can't function at all without relying on the representations from System 1 thinking (which are formed in pretraining).

So you could argue System 2 thinking is some part of the LLM, but it can't be 100%.