Oh wait. I lost all respect to those hype-men who claimed it's just around the corner.
Before transformers, people claimed that you just need bigger neural network, not something new. It did not work.
Then came transformers, the new thing can again scale up better than previous era. Transformers have obvious limit: the computational complexity of self-attention module scales quadratically with the sequence length. We are trying to get partial fix with sparcity and linear tranformers and other techniques, but it gets harder at every turn.
We make progress, but "just scaling" is not working.