We aren't going to see ludicrously high clock rates for the foreseeable future. There are a lot of compounding factors as to why, but the biggest ones are the pressure for efficiency driving designs that aren't dumping higher and higher voltage to get frequency, the diminishing returns on voltage vs frequency (see Ryzen, where a 20% improvement in clocks costs about a 50% increase in power draw across all skews, and similar situations happen with Intel).
That being said, a 4ghz Skylake core crushes a 4ghz Core 2 core. Depending on your benchmark used it can perform anywhere from 80% to upwards of 170% faster per clock. You don't get as dramatic year over year improvements increasing the the per cycle performance, but innovations leading up to ~2004 (or 06 for the multicore boon) were just stuffing power hungrier and hotter transistors on smaller dies.
Non-x86 systems went over 5 GHz on air many years ago, e.g. IBM POWER.
Core 2 Duos were slower per clock than Pentium 4s, I think by quite a bit. It was a real set back for performance when they came out.
Core 2 was the breakthrough that left AMD in the dust from which it still hasn't recovered to parity. Even if I can't recall the numbers, I fail to remember how the very energy-intensive (high clocked) Pentium 4 could have been faster per clock.
Are you sure you're not thinking of the Pentium 3 to 4 change? After all, Core architecture had more in common with P3, didn't it?
i5-4670K AT 5GHZ OC On a $30 Air Cooler https://www.youtube.com/watch?v=NUHm2qHI3gc
I really wish we could see more IPC and 10Ghz+, our single threaded performance have be stuck at the same level for far too long.