You're not going against the will of God by stopping at 3. Or by not having children at all.

But if you give contraception a fixed place in your life, to have sex which is fundamentally closed to children, then you have introduced a contradiction into your body and into your life.

And yes, He rules over 100 billion galaxies (more if we live in a multiverse) and cares about such things. If you think this is petty or spiteful, then you think too little of your sexuality and of yourself. These 100 billion galaxies were required for you to exist as a human being, called to (try) living a life that prepares you for Heaven.