Another thing not yet mentioned is that it literally becomes physically more difficult to have children as you age. Female fertility starts to rapidly drop in their thirties, and many will hit menopause in their 40s. The exact age is somewhat random, and some women will even enter menopause in their 30s.

Before I had children I thought it would be relatively easy - that's why you use birth control after all. But when you actually have children you learn things like at best you're looking at, at best, a 10-30% probability per month if you hit the ~48 hour ovulation window just perfectly. That doesn't sound so bad - because a month isn't such a long time, and ovulation is pretty predictable. But when you start late each month matters, and then if you want to actually have multiple children, then you're already looking at a years long process.

And then add in that as you age, all sorts of birth defects and disorders like Down Syndrome become much more likely, and you can't effectively test for them until about halfway through the pregnancy. It's just not a great idea to start late. I'd also add that for us to have a sustainable population, everybody needs to be having more than 2 children on average. This is going to take a pretty substantial reshaping of society and culture, or our society and culture will simply go extinct.