My (perhaps naive) understanding is that, for Spectre to be truly exploitable, an attacker must be able to execute code locally (I know of the JS POC but there are browser mitigations). But if an attacker has an account in my system I'm already screwed anyway, as there are probably other privilege escalation vectors that are easier to exploit.
My laptop is not high-value so I'd rather just disable that stuff. I once timed a C++ build, with and without Spectre mitigations, and the difference was something like 20%.