Chicago neighborhoods on the near north side (which are some of the most walkable in America) have large supermarkets. They just have parking garages (either above or below them). They are slightly smaller than suburban ones, but not that much.
Whole foods has plenty of them in all big cities.
It's just a cultural reason why some cities lack them--probably because people don't cook at home as much in NYC, for example.