Sure, lots of things affect your quality of life, some slightly and some greatly.

However, the job market is qualitatively different, which is why it should be held to different standards. Loneliness is definitely harmful, but still many people don't want sexual/life partners. However, everyone needs to eat and have some sort of shelter and access to health services. If you lack money for this, you either die or become a burden to someone else, possibly the state.

Coming back to what I believe was the parent's point when they said "in America the market determines your self worth", your worth as a person isn't -- shouldn't be! -- how much money you make or even whether you have a job. That simply doesn't seem right to me.