You don't need to be relevant, just being accessible is enough reason for all sorts of bots, legit or sketchy, to shove hundreds of thousands of requests down your throat.
54 karma · joined October 1, 2014
lately? more like for the last 10 years afaict.
User-Agent: bender
Disallow: /my_shiny_metal_assIt's still my favorite tech talk, very fun to watch.
a company known for their respect of privacy and exemplary business ethics. good that he left facebook.
It's even considered harmful since extra domains means extra dns lookups which could take quite a while.