I run my development stack on a few VMs and honestly even with Vagrant or others solutions the overhead/resource usage is noticeable and it doesn't feel that nice. It's fine for dev stuff though.
I imagine you're suggesting something like X over SSH, right? Do you use something similar on a daily basis? I would probably commit sudoku if I had to use the entire VM (not headless).