I agree wholeheartedly. For comparison, I currently see ~95% of search engine referral traffic from Google, 2.8% from Yahoo, 1.4% from Bing, and <1% from all other search engines combined. It makes no sense to let X, Y, Z random search engines index content like Google, when they send nowhere near the traffic. As others have mentioned, blocking by robots.txt often doesn't work as some bots don't obey it. I currently use an extensive rewrite rule to block those bots' User-Agents, and block specific IPs as a last resort.