But the argument is that by dint of repetition, the models actually encode some of the structure of basic logical primitives: syllogisms, entailment, modus ponens/tollens, etc.
This then weights their output so that (for simple enough stuff) they're more likely to emit outputs that are logically sound, than outputs that aren't logically sound. Indeed, this has to be the case, or they couldn't maintain any level of coherency in their output at all (which, like it or not, they can.)
Like you, I'm not comfortable calling this reasoning. But it's also something that is not 100% entirely unlike reasoning, either, at least in terms of output.