Socialization. For us, this goes beyond the normal bit where your kids don't spend as much time interacting with other kids and learning how to live in a society. We live in France, so the local village school is the perfect chance to toss the kids in head first to a new language. Both are swimming nicely, so it'd be a shame to pull them out and lose that before they're fluent for life.
Stigma. In the US, it seems that homeschooling is what crazy bible people do to shelter their kids from the real world. I don't want to set our kids up for a lifetime of having to explain themselves to everybody they ever talk to about education.
But the upsides are pretty tempting. Work-wise, I've been 100% remote for a long time, so the only thing keeping us from traveling the world instead of living in a house is having the kids in school. I expect we'll try taking the kids out for an entire year before long and booking a one-way flight, just to see how well it works...