So true! Search engines in a way have both invigorated, and ruined the web. They are great because now you can find anything, but they suck because they've removed a lot of the mystery.
I loved hearing about cool websites via word-of-mouth. Starting there, and then just following a rabbit hole of links to places I've never been before.
I suppose it's still possible if enough people agree to host sites that reject crawlers.