As for boom and bust - agriculture is like investing. You may not make much in a year, and you may lose money in two or three, but over the course of your lifetime, you accumulate wealth (land) and prosper. The issue, at least in my area, is that property is just prohibitively expensive. I am by no means in a hot area of the country, but the price of an acre of land has almost tripled in the last 15 years.
Also - that so much of our food is produced by so few individuals is an incredibly recent development, like less than 40 years recent development.
I'm not really sure what all that adds up to, but I'm sure it's something.