This works when everyone owns many acres of land per person, and is spending their entire life working the fields, doing hard labor, and doing very little else with their life yes. We don't live in that world anymore, though.
I suspect there isn't even enough physical land on the earth to support this inefficient method of farming.
We have significant evidence of how this worked out for people. That world that you are describing, where everyone spent their entire life just trying to barely feed themselves, was a horrible place, for everyone.
This period of time was called "the history of the world before the industrial era". And lots and lots of people died. So no, they were not fine.
> It is a reality for lots of people, and we're pretty happy about it.
By "a lot" do you mean a very small percentage of the total population?
> My tomatoes cost me 2 minutes of time.
What you do with your tiny backyard garden is irrelevant. It is mathmatically impossible for you to be feeding yourself entirely on that, unless you have multiple acres of land, which I doubt is what you are describing. Your anecdote does not overrule physics.