Everyone says "those autoregressive transformer LLMs are obviously flawed", and then fails to come up with anything that outperforms them.

I'm not too bullish on architectural gains. There are efficiencies to be had, but far closer to "+5% a year" than "+5000% in a single breakthrough".

You can try to build a novel AI architecture, at a small scale. Just be ready. This field will kick your teeth in. ML doesn't like grand ideas and isn't kind to high aspirations.