thoughts just show up and we notice them after, so imo we're the observer, not the doer. LLMs do the doer part well by mirroring human data, but there's no observer.
I feel building metacognition into current architectures might be what gets us to AGI