Each process has a set of page tables, which describe how physical memory is mapped to their view of memory. The entries in those tables can be marked "not present", so that when they're accessed by the process the kernel is signaled via what's called a page fault.

The interesting part is that this is used for both invalid memory areas (which cause segfaults) and for virtual memory. The kernel can take memory your process hasn't used in a while, write its contents to disk, and then mark that area "not present." Then it can give that memory to someone else, and load your data back in when you try to access it.

This trick is also used to load in binaries and other files. Instead of reading a program in all at once, the kernel just updates some internal bookkeeping to say "these pages should be from this file" and then lets the page fault handler load them in on demand.

The problem here is that the actual kernel page fault handler is plenty fast, but the hardware mechanisms that set it off and finish it are slow, because of how the CPU is built.