Wouldn't it make context switching faster because you don't have to invalidate any TLB entries?

Context switching is more efficient due to less context.