Wait I thought distros ran one X server per logged in user account and put them on different ttys. Out of the box on Ubuntu I can log in as two separate users simultaneously and switch between those unlocked sessions as well as the login screen by changing the tty.

If you pass the GPU device node through to an LXC container I believe you can run an X or Wayland server fully inside of it but I'm not sure if changing ttys will work correctly because you might need access to more than just the GPU to handle the handoff with the kernel correctly.

I don't intend to run more than one X server. Most of my GUI applications don't use X.