Mind you, in the end, I still think the balance favors some very simple expansion over the course of thousands of years, but there really are an awful lot of forces inhibiting it.
Also, the movies made this solitude vastly worse. The books reveal that JRR Tolkien was aware that all the cities would be surrounded by farms and villages and be nations. The nations may be disconnected, harried by the aforementioned evil, and find it very difficult to expand, but he understood there would be infrastructure there. Movie Gondor is ludicrous nonsense; able to field a significant army and sustain a civilian population, but apparently the army and civilian population do not ever need to eat anything. Or the food just beams into their granaries from space. Or maybe their Numenorean background or Rohan's affinity with horses means they can eat grass. Because they don't have any farms or any other infrastructure. It's city, and then literally 100 feet outside of the city walls, you're in the wilderness, effectively as untamed as 100 miles away from the city. You might justify it on stylistic grounds, but it has no other justification.
In the end you have to grant all authors grace. Nobody can run a full world simulation in their head accurately. If this is all we have to grant to JRR Tolkien, he asks for less than many other authors. Net-net I still think the wildernesses in his world are unrealistically sparsely populated, but it's far from the worst idea I've swallowed to read a book.