Just had to use Firefox remotely (to use WeTransfer to ship a giant file, they dropped support for their cli, grr), and had exactly this issue - it works incredibly slowly over X over SSH. In the end had to use vnc over SSH.
(Note that tightvncserver seems to silently ignore the -localhost option now, which means it's completely insecure to run on an internet connected machine. Tigervnc still has the option)