I'm not nearly as worried about the 3 R's. I just don't expect that we'll find there's much long term difference between kids who are taught to read in 1st and second grade, and kids who are taught to read in kindergarten and 1st grade.
I'm not sure how much school has changed, but for me, in the 90s, the ~90 minutes (split into three uneven segments) per day of physical activity was just not enough. Could have used half and half.
I think it's "probably good" for them to learn something, too. It's not as if it's a 9-5 job after which they come home, do housework, file taxes, try to socialize with the family a bit, and go to bed exhausted. They do still have time to be kids besides school at that age (barring less common cases like toxic families and extreme poverty, of course).
Also, we didn't evolve to read books, use chairs, power our stove with electricity from hydroelectric power plants, etc., but I'd rather not cut those out of my life either. We did evolve to run many miles behind prey, but I don't mind not practicing that. It's a really weird argument that a lot of people seem to use, "we didn't evolve to", which can be applied to virtually everything we do today. It seems to be a placeholder for when something seems disagreeable to you but you can't actually articulate why or be bothered to look into whether that is the case.