The farms next to me (I live rural) are 20-30 acres and are farmed by professional subcontractors that employ dozens of (immigrant seasonal) labour for picking etc. and use heavy machinery that costs hundreds of thousands of dollars and are shared between multiple farms.
Likely if your old boss operated a 'micro' farm it was only using a fraction of that acreage.
Also 3 acres is not a "garden", it's a very labour intensive high production "market garden" and could in fact be called a farm of a type. That's 360x360 feet, and maybe you can't visualize that, but it's large. It would require machinery and hired labour to manage. No single individual could plant weed and cultivate that single-handedly. My garden on my acreage is 100x100 and if I could actually keep up with it would easily produce enough for 5 or 6 families.
I'm not saying it's a profitable farm, but it's a kind of farming. You could easily produce $10-$20k of revenue out of intensively market gardening something of that size.
But even more if you were to do cut flowers or something higher profit. My neighbour is very profitable with lavender and has about 3 acres of it and makes quite a bit of money off just that. But he is connected deeply to the flower business, so knows the game well.
A 3 acre high quality vineyard could produce $10k in revenue just selling the grapes where I am. If you made that into wine (capital intensive and regulatory hassle and not allowed on that small of acreage, but still) you could actually make many multiples of that and make a good living once you covered your upfront costs.
So yes, it's possible to 'farm' on small acreages and make an 'ok' living if you don't mind busting your ass. Just not the stereotype of what people think of as farming (coy/sorn/wheat/sorghum cash cropping, etc.)
Myself I have a bad back now, so no thanks on the pay cut, I'll keep these things as a hobby :-)