It's not just a problem of private schools - it's families having the ability to move to a different district.
If you're affluent enough, no one can force you to send your kid to a given school. They want to bus your kid? Move to the suburbs. They want to bus from the suburbs? Send your kid to private or religious school? Ban those? Hire a nanny/tutor. So forth and so on. As you pointed out, there's not benefit academically for these advantaged kids, so this understandable why there's push back. Some people aren't going to value being exposed to other communities enough to outweigh the inconvenience of busing their kids.
We could try to make these communities more desirable to live in, but 1) we're assuming we haven't tried making them desirable (and if not, why not), and 2) we risk pushing out the poorest of the community and negating the benefits for those families. It's a big, hard problem - how do you go about improving a failing school in a poor neighborhood without changing the neighbors in the neighborhood?
I was too young to remember the 70s, but, from what I understand, busing occurred mostly between neighborhoods in the city, often moving kids from one poor community to another. If you were in a neighborhood with nicer schools and busing was enforced, you moved. And the politicians and judges enforcing busing certainly didn't send their kids to these schools - if they lived in these communities at all.
Unless we start telling people where they can live and how they can raise their children, I don't think this path is viable.