That works both ways. If you believe parents shouldn't bring their children to the office space because they should respect you and your desire for children not to be there, then it's only reasonable that you should respect their desire to have their children with them. Whether you're fond of babies or not is irrelevant; the important thing is the impact on the business. If having children around means an overall productivity drop then they're a net loss, and a dedicated child-care facility is a better option. But if the parents in the office are more productive with their children close by, to a point that outweighs the loss of productivity from the non-parents, then children in the office is a positive. Few workplaces have ever measured that, so there isn't a simple "it's worse with children around" answer.

Plus, and this is really important, there's a distinct danger that employers who say they don't want children around the business using that as a way to limit opportunities for women (albeit subconsciously; few would admit to it). That's something to guard against.