I do not think it's a disingenuous way to frame the challenge facing people who live in the area. Two-bedroom apartments should not be the privilege of those in a certain stratum of society and above. They should be something that is within reach of nearly everyone. Nor should housing in the San Jose area become the domain of a privileged minority.
You unfortunately have to find a balance between location and luxury.
Where do the people working in that Walmart live?
As to your first point, 5th Ave is smaller, but also more expensive. SF is bigger, but relatively cheaper. It's a sliding scale; both driven by scarcity of land compared to desirability.
New York has a working mass transit system, though. For poor people this makes the world of a difference. I remember that, when I lived in Madrid a few years back, I was impressed by how fast you could go _everywhere_ in that city with the subway alone. A cross-town morning commute in that 3Mil Juggernaut took about as long as the cross-town car-commute in a less developed 130K city I had lived in before.
And actually there are more affordable rooms in commuting distance (BART or car) than it appears from popular opinion online.
Either buildings can go up, or prices. They're not making any more land.
Edit: In retrospect, I made a critical error; I can't tell if the downvoters understood I was loosely referencing Lake Wobegon, or if they thought I was just dumb