I was thinking about this setup as well, but wasn't sure if I was nuts. What setup are considering? Plug a hefty quad core tower in at home and then use a chromebook as a remote control with a screen?
I don't use Chrome Remote Desktop though, just VNC or Vmware Shared access through Crouton on my Chromebook.