These things are just statistical language models, aren't they? To the extent that when people reason or plan and then verbalize those plans, they tend to leave certain trajectories through language space, a statistical model could presumably reproduce similar, plausible sounding narratives. Doesn't mean that any kind of agency ever actually thought about those plans though, in the way that a person would.