Not an expert, but if you are running absolutely no untrusted code (which is hard depending on your definition of untrusted), then the risks are low.

However, when a vulnerability is found, then spectre etc. make it easier to abuse that vulnerability to do something useful.