You leave so much performance on the table if you demand cryptographically secure perf for everything. That means no caching, everything runs in fixed, worst case time, and the amount of work must be fixed as well so there is not any variation in power draw.
Here is a simpler solution - do not run untrusted code on your computer. It is just that easy™.
PS: I disable JavaScript by default in my browser, and manually whitelist exceptions, for the same reason. I also turned off Spectre mitigation on my workstation.