20 years ago we had Pentium 3 500MHz, 2 FLOPs/cycle, 1 GFlops.
GPU of my desktop PC computes 11 TFlops, that's 4+ orders of magnitude faster.
For CPUs alone it's indeed less impressive, $500 for that P3 inflated to $760, the comparable modern CPU is Threadripper 2920X, 16 flops/cycle * 12 cores * 3.5 GHz = 672 GFlops.
But they’re much lower quality (less flexible) FLOPs. If you don’t have much data parallelism or heavy conditional logic for each datum, you’re not going to come close to 11 teraflops.
Update: also, for other tasks, GPU flops are of higher quality than CPU.
VRAM bandwidth is ridiculous, much higher than system RAM, computations that are bound by RAM bandwidth will be much faster.
GPUs mask RAM latency with hardware multithreading, if your code is parallel and needs to access large amount of data randomly, a GPU will saturate these flops, a CPU will sleep most of the time waiting for data to arrive.
CPUs can be faster for RAM bandwidth-bound code, if there's not that much data and it fits in caches at least L3, CPU cache bandwidth is very good.
GPUs can be faster for heavily conditioned code. If it's the condition on RAM value, a CPU will stall or mispredict branches, both slow, a GPU will switch to another hardware thread which is very fast.
Despite these differences, I find flops quite useful in practice to compare performance of different computers across architectures of generations. Computers compute, flops measure how fast they do.
Using double precision SSE scaler ops, the Pentium III could execute one addition and one multiplication per cycle. (The throughput was the same with vector math because the Pentium III only had 64-bit SSE units. So a 128-bit packed multiply and a 128-bit packed add, four double precision operations, executed over two cycles.)
The only difference is that CPU computes stuff on 4-wide vectors of these numbers (modern CPUs up to 16), that GPU on 32-wide vectors (other modern GPUs up to 64).
But the comparison is against an old CPU, without big vector units.
Those non-vector calculations are very different, and far more flexible.
A brand new CPU core can do 3-4x as many separate operations per cycle, and is clocked 6-10x as fast.
Having more cores helps but it's still far behind.
Also for a fair price/performance ratio you probably want to compare to the 450MHz model at $230, so only a $350 CPU today ($280 equivalent by august). https://money.cnn.com/1999/08/23/technology/intel/
They’re both very flexible, just different.
For example, GPUs are better at vectorized condition code. CPUs were mostly fixed with AVX512 but these instructions are too new, only available on some servers.
Sure, there’re algorithms which don’t work on GPUs. A stream cipher would be very slow because requires single-thread performance, also GPUs don’t have AES hardware. A compiler is borderline impossible because inherently scalar and requires dynamic memory. Also GPUs don’t do double-precision math particularly fast.
Still, I think many users who need high-performance computing can utilize GPUs. They’re trickier to program, but this might be fixable with tools/languages, we have been programming classic computers for ~70 years, GPGPUs for just 12.
That's throughput, not flexibility. I would define flexibility in terms of how easily the instruction stream can vary per math operation. Full flexibility requires a lot more transistors per FLOP, which is why you can't use wildly different architectures to assess Moore's law, which is about transistor count.
And comparing transistors on Pentium 3 (including cache) to an RTX 2060 (including cache) it seems to be 34 million vs. 10800 million. That's two and a half orders of magnitude.