You might want to be careful about anachronism when stating facts.
It is true that very few today farm, but take into account that, within the past century alone, the yield of farmers has increased by an order of magnitude, not to mention the millennia of domestication and invention before it. For most of agricultural history, the percentage of farmers was much higher than 3%.
For example, from Wikipedia's article on Roman Agriculture:
>In the Roman Empire, a typical family of 3.25 persons would need between 7-8 iugera of land to meet minimum food requirements (without animals). If a family owned animals to help cultivate land, then 20 iugera is needed. There would not be a surplus production on this farm.