Hacker News Reader

Comment by alcari | Hacker News Reader

Not necessarily, because things like robots.txt will still prevent it from ending up in search engine results.

I forgot about the part where it's mandatory for search engines to comply with a robots.txt

Well actually the relatively hard part is hosting a crawler of decent size, and then if you crawl in violation of robots.txt its pretty straight forward to use iptables to ban you, of course you then spend money on hiring a botnet to mask your traffic footprint, except that on that same darknet there might be people who are friends of the owner of that botnet.

It is non-obviously difficult.

Does this not rather beg the question - is there a deep web, and how big (or small) is it. I can easily understand the desire for a coherent group of people putting up vpns etc to keep their world seperate from others - but that implies you join in based on some other criteria, which sounds not very deep web but pretty secret-VPN-we-are-not-telling-you-about-unless-we-cross-your-AS-Number-when-something-is-obvious

It just has that feel of "secret society" to it, which tended just to reflect the informal power structures of the wider world anyway.

No, the "deep web" is real in the sense that there are billions of network addresses that contain content or services which are not accessible through the 'standard' discovery services (Google). In many ways things like Usenet are still part of it as there are netnews groups, and they get used, but there isn't a lot of indexing going on. Further there are at least two 'separated' NNTP type networks that are invitation only.

So it is a "collection" of secret societies, each with their own quirks. As a collection is constitutes a 'web' and perhaps the only commonality is the desire to not be part of the "public" web.

[1] http://begthequestion.info/

Reply on news.ycombinator.com