The only way to access Silk road was through TOR
325 karma · joined January 29, 2012
Web scrapers look just like real humans, and its publicly available data, so when a website doesnt authorize web scrapers, how do you draw a legal differentiations?
behavioral modeling - rate limiting, bandwidth restrictions, etc
identity verifications - make sure they are running the browser they say they are, allow google and other search engins by whitelisting their IPs, block others that are pretending to be google, etc
code obfuscation - make it hard for them to scrape your code. Change up the CSS, etc.
OR you can use an automated service to do all this for you. Check out www.distil.it. Full disclosure, I'm the CEO of Distil.