X11 forwarding doesn't cause any problems? I imagine I would easily be running 6 sandboxed apps at the same time.

From what I heard you can escape docker anyway so I would need something like SubUser (suggested by amouat)