But the alternative is I could end up in the ICU with pneumonia without it (and have, in the past, from the flu), so the risk/annoyance is worth it.
Just to be clear I do HIIT, strength training, and (my favorite) lots of yard work regularly and the flu vaccine doesn't interfere with any of that.
I don't know how common it is to experience this, as I would say I have a subpar immune system (frequently sick as a kid, history of food allergies, etc.). But when I've discussed it with doctors they do say my reaction is unsurprising, however the benefits outweigh the "costs".
In January, this would have felt like a terrible deal. Right now, it feels like you're telling me that - should I socially isolate for a few weeks a year after vaccine shots - I can largely go back to living a normal life. That feels great.
As long as it's not a Grade 3 fever, I don't see a problem: many vaccines can cause (transient) fevers. (Part of that is due to the adjuvants used to prime the immune system by causing a localized inflammation).
I got a shot for the yellow fever a few years ago and I felt quite bad for a couple of days. When I was asked for a booster shot for measles, I was warned of the possibility of developing fever for a couple of days as well.
Discomfort is totally expected with vaccines.