Cache is not fast because it's physically close to the core. Well okay, closeness does matter but it doesn't matter as much as the fact that cache is static, i.e. it's (effectively) made of flip-flops.

If you build a computer with DRAM 10 cm away from the CPU and static RAM 50 cm away, access times for the static RAM will still be much lower than access times for the DRAM.

Why don't we just build all RAM as static? Because it's more expensive per bit, it uses more energy, and it requires a great deal more silicon area.