Bad argument. You could have had even more children, is the fact you didn't immoral? What kind of monster are you?
However, I wouldn't push the point. My view is that "life is good so having children is good" but not "you must have as many children as possible." There are various ways I might justify this philosophically, but I would just say that parents have a special obligation to their children and, if you have too many children, it's impossible to fulfill that obligation. So you should have as many children as you can effectively parent.