NYC is an archipelago separated by large bodies of water, and it is hard to avoid going through Manhattan to go between the western suburbs, the northern suburbs/Bronx, and the eastern suburbs/Brooklyn/Queens. What few links exist are always horribly congested.
Geology is the reason there aren’t skyscrapers in the west village etc though.
The West Village is actually not very well connected transport wise, and historically the waterfronts were docks-oriented. Downtown is the historic core where pre-bridges, all the ferries went, and where many bridges still exist. Midtown is the location of the rail terminals, so a suburbanite can walk to their job without having to pay a second fare. (IIRC Grand Central is where it is because at some point the city banned surface-level steam trains below.)
Downtown is where all the ferry landings were, and where a lot of the bridges were built after.
Midtown is where the rail terminals stop, so suburbanites can quickly walk to their jobs instead of paying a second fare for subway/bus/taxi.
There is no correlation between bedrock depth and height in Manhattan, as explained in the article of the grandparent comment. In fact, most of Downtown is landfill.
I think your causality is backwards. Why has downtown and midtown always been the two main population centers, and not one continuous city? Because that’s where the exposed bedrock is, with some swampy midlands in between in the West Village. Geology.