Well yeah, if you enjoy doing Activity then the prospect of having to get a new job and therefore spend far less time doing Activity might be understandably upsetting.

If you can't get paid to think about math anymore then you are by necessity going to spend a lot less time doing it.