I have a quad-core tower with 12GB of RAM and an SSD, that primarily runs a Windows 8 and Xubuntu VM. Performance over the local network almost feels like bare-metal.
I don't use Chrome Remote Desktop though, just VNC or Vmware Shared access through Crouton on my Chromebook.