Rather than upgrade to an SSD, though, I ended up replacing it with a used i3 NUC. (Partially because I wanted to try out the HDMI CEC adapter from Pulse Eight, and partially because the premium from selling the Mac Mini covered the cost of a faster NUC + SSD + CEC adapter with a little bit left over.)
BUT I didn't really use the CEC parts that much, and the Intel sound drivers would "loose" my speakers and start refusing to output sound over HDMI anytime the TV went to sleep, forcing me to reboot the entire computer to get sound again. I ended up giving the NUC to my dad. (The drivers don't seem to exhibit the same bug with his monitor.)
I'm now onto a compute stick + USB sound card (because it had the same HDMI audio bug as the NUC), and I still occasionally wish I had just stuck with the Mac Mini.