If you remove parking spaces it solves itself because traffic is reduced and transit options become more efficient AND more financially sound.
But if people in the main stop going into the city you'd probably see a drop-off in the city amenities that make many people want to live there in the first place.
Nowhere in the world, and I mean the entire world, has the scenario in which surface parking lots are replaced with other productive uses have resulted in a drop off in city amenities - it’s a non-sequitur. The businesses and residences that replace the lots are city amenities. Adding them has the opposite effect that you describe.
Think about it another way - what if we add surface parking lots? What would you drive to downtown to do? There wouldn’t be anything there because the amenities would have been replaced by mostly empty parking lots.
We can also just have multi-story garages. We can actually increase parking (on a social scale) while removing surface parking lots. That would create amenities and allow folks like yourself to easily come to town. Would it cost? Sure. So what?
Just go (or don't go) to places based on how much of a hassle it is to do so. If enough people--local or otherwise--want to visit good for them. I definitely make choices based on how easy or hard it is to get to the destination.
Which means you also need to battle the housing problem, too, though, plus changes in settlement patterns take years to decades to manifest. In the meantime, you might have to weather quite some griping about it or even serious pushback.
I don't think society needs to accommodate that lifestyle so someone can live 30 miles away from work and treat my city like a place you just commute to work to. Those days are increasingly over, as cities realize this is bad for the city and incredibly expensive to operate (surface parking lots are economic extractions and tax revenues low).
You are of course right there is pushback, and things take time to manifest, but we moved to the suburbs at one point there's no reason we can't fix that. I'm not entirely sure why my city council for example cares what suburban voters who don't vote in our elections really think outside of 2nd hand complaining from employers. But they're free to relocate their large downtown offices to the suburbs, we shouldn't cater to them anyway precisely because they can move at any time leaving quite the financial problem as building patterns revolve around this 8-5 white collar commuter scheme to surface parking lots, and if the anchor tenant leaves you're left with, basically, a dead city. It's incredibly fragile and stupid.