"Afford" is said in many ways. I know my kids will probably have to take out student loans to go to school. Other than that, you live frugally. Health insurance costs usually cap out at 3 kids.
It's not exactly the incredible expense some folks imagine it to be; it's certainly not worth not having children over.
Unless of course you're dead-set on sending your children to Oberlin or Harvard or something and coming out of it with no debt, I don't think it's a reasonable objection. And even if you were dead-set on it, shouldn't you be making enough to be able to afford to let your wife stay at home if she wanted to anyway?