The basis of the U.S. "yeoman farmer" idea was, in part, that there was land available for continual expansion. Simply go west, kick out the natives, and all your citizens could be given a plot of their own - organizing around this goal gave the government stability and purpose, even though it had several obvious flaws.
This was also the genesis of the internal conflicts during the 19th century; the eventual victory of the federalists in the Civil War was, in effect, the creation of a new nation organized around industrial capitalism.
Which is why the war was started by plantation slave-holders: They knew that the only way to keep their slaves was to dissociate their economy and politics from that of the industrialized North. Even after they destroyed the concept of states' rights by passing the fugitive slave laws, they still weren't confident that slavery could endure when the North was going great guns without it. When the North proved that they could elect a President without pandering to Southern Slave Power interests by electing Lincoln in 1860 without a single Southern vote going to him, that was the last straw.
(The election of 1860 was, in effect, two elections going on in parallel: Bell v Breckinridge in the South, Lincoln v Douglas in the North. Except, of course, only one man could be President, and Lincoln won the Electoral College by a convincing majority.)
They didn't mind the size of New York or Philadelphia. Cities were very small in 18th-century America. In 1790, New York City's population was only twice that of Charleston.
The reason for DC's location on the Potomac was that the Southern states wanted the capital to be in the South.
The temporary capital of Philadelphia posed an additional problem, in that it was located in a free state. George Washington had to rotate his slaves through Philadelphia while he was President, so that they would not stay long enough to be emancipated.