It's a bit concerning that we are not really making amazing progress in single threaded performance.
I think we might not be going quite far enough with complex instructions, and maybe we need to be able to define out own with custom microcode or something.
Like, a CPU that had just a few instructions, enough to bootstrap and that's it, and a very fast FPGA that could switch configuration quickly to make your own instructions.
Right now it takes time to send stuff to the GPU. We need acceleration we can access in one instruction, so that normal compilers can generate it all behind the scenes.
Or, alternately, modular pluggable acceleration. You could fit like 8 little MicroSD sized special purpose accelerators on a laptop MB.
Maybe file serving can be hardware accelerated. Is there an actual reason you need a beefy server to serve a ton of media, or can CDN caching be optimized the way ethernet switches are, so you have a tiny credit card size device serving 10gbps on three watts?