The anxiety really comes when we start thinking about old age. Where dipping back into the workforce if we needed to is not really possible and unexpected expenses are likely. So I guess it's keeping to the grind to build as much of a safety net for those days as we can until we really decide to pull the trigger.
We both know enough people who are "independently poor" and seem to make it day to day without too much fuss in the short term. But almost all those same people we know end up in catastrophic financial emergencies over the longer term due to unexpected expenses they can't surge their finances to meet. For us it just means we don't put as much away for retirement for a paycheck. For them we've seen evictions, garnished wages, divorce, ruined credit, bankruptcy, and so on. It's a vicious cycle that starts with being relatively expensive and ends up being ruinous.