I don't think this is a real problem IMO. If you learned it in the first place, you can learn it again after slacking off for a bit. You might need a few months for the focused effort, which is honestly not a cost but probably what we in the industry call a sabbatical.
It may add some risk to getting a job if your life is suddenly turned upside down after slacking and you need a job right this instant. That slacking might fail you some interviews, but not all of them. Even in this situation, there are financial options and temp jobs to get by with.
I don't think anyone should ever be convinced that if you stop practicing something that your ability disappears completely. I'd put some faith into the idea that you can relearn it just as well or better when the need arises.
Doing less on the clock and doing less in FOSS/personal learning are two separate categories. You can do less on the job without inhibiting growth by using that energy for personal work