I think this is upside down.

AI is actually still really bad at architecture. It produces very good localized code, like an auto complete. It produces so much of it, that it can be overwhelming, but it's not remotely better than the equivalent aka an infinite number of talented humans doing the work.

The scale of it (aka 'make me a video game') is daunting and I think misinforming our intuition as to what is going on.

LLMs are brute forcing outputs and attacking math problems is not a singularity, it's just more compute.

What we are witnessing is the ability for machines to use fuzzy concepts, instead of very strict code, which allows it to map into a much greater variety of things, aka code, pdfs, web - and crunch/compute the content.

It's just compute.