This is very true. My scraping efforts have become vastly more sophisticated after running into explicit attempts to block me. Now I've got all kinds of bells and whistles, and validate the data returned.