You are correct that your argument is a common one. There is also a common response - the invention and use of artificial fertilizers allowed us to dispense with good land use practices, hammer the land with chemicals to squeeze as much crop out of it as possible, and dramatically raise the amount of crop grown for meats and industrial outputs like corn syrup and ethanol. We have way more crop land than is required to feed our population a healthy diet, we are degrading our soil, and modern farming economics have become so dependent on industrial inputs that farmers in India have committed waves of suicide after becoming so indebted to Monsanto/Bayer that they are unable to imagine any future for themselves.

You are right that we are so dependent on artificial fertilizers that we cannot simply cease production tomorrow, but that only speaks to our dependence on them due to economic and social factors, not their inherent necessity to our ability to feed ourselves. Essentially no one is saying we should shut down all the factories tomorrow so the argument is a dead end.

What many people are saying however, is that we went very far down the wrong road, and we are paying the price today. We must figure out how to bring our farming practices back in to balance with ecology.