Sharing an address space is nice. Programs pass lots of pointers into the kernel (think read()/write()). For the kernel to get at that data, if the kernel and userspace share an address space you can just use the pointer. If userspace has a truly separate set of page tables, you have to do some relatively crazy (and slow) stuff to go and access the data from inside the kernel. It's much harder than just accessing a pointer.

We can still do this with KPTI because despite having two page tables, the kernel still continues to map all of userspace.

BTW, the bit flipping is pretty much

// Allocate 8k which is also 8k aligned: pgd = alloc_pages(PAGE_SIZE*2);

That allocates something which might go from (physical address) 0x12340000->12360000. The first top-level pagetable is at 0x12340000 (for the kernel) and the second is at 0x12341000 (for running userspace).

So, when you leave the kernel you just set a bit in your CR3 register (well, you do it in assembly, but here it is logically in C):

write_cr3(read_cr3() | 0x1000); // 0x12340000 -> 0x12341000

When you come back in to the kernel you clear the bit:

write_cr3(read_cr3() & ~0x1000); // 0x12341000 -> 0x12340000