- Using DNS "zone files", the DNS database for TLDs (which are not available for all, but most) show there's circa 200 million domains registered at any moment
- A large percentage of these are parked, i.e. no unique content.
- Many domains are "tasted", i.e. bought, are alive for a few days then disappear, so potentially you waste time crawling them
- Lots of sites are database driven and can result in millions of pages that can be created in a day
- URL rewriting means you can have an almost infinite number of pages on any one site
- Soft 404s and duplicate content can be hard to spot and can waste resources in gathering/removing them
There's paid for resources like Majestic/Ahrefs/Moz that crawl the web to see who's linking to who and they all contain trillions of URLs.
I think the most detrimental fact is that pages often disappear or change, I don't have a recent number but I'm fairly certain there's a 10-15% chance that any link you see this year, will be gone next year. "Link rot". Hard to build a DMOZ style directory on that scale with that problem.
I don't think it is unmanageable, it just needs to be seen from different perspectives and managed by different groups of people.