That “after a few years’ experience” bit is the catch. What’s happening during that time is a whole bunch of skill acquisition, building mental domain models, building context- and culture-specific soft skills, etc.

All of it relies on learning and information processing capabilities that LLMs simply do not have. The idea that any of it will automatically emerge from the current architectures simply by adding more parameters is about as sensible as thinking that if you keep counting long enough eventually you’ll get to complex numbers. We’re going to need to invent new technology to fill these gaps, and there is no way of knowing how long those developments will take.

It is unclear to me that "transform society" is automatically a good thing. Nuclear war would transform society too.

"Beseech the god machine that it may shower upon you its tokenous beneficence."

The god machine can prompt itself. Just go "Write me a prompt to <whatever>. Ask me as many clarifying questions as needed.". It is not a differentiator.

Deep domain knowledge and pattern matching that allows you to quickly verify outputs might be a differentiator. Maybe. But you might end up the bottleneck if you verify everything.