To the best of my knowledge, one downside is the text rendering is synchronous. Read a character off the tty, try to display it on the screen. It's fine until you go and cat /dev/mem or something like that. Many GNOME-ish terminals use vte which drops frames if it can't keep up rendering.
I also run into problems with bad display managers or DEs that don't load ~/.Xresources or ~/.Xdefaults (looking at you Debian gdm).