It's not like robots.txt is a great deterrent for crawlers. Only reasonable barrier would be a paywall hiding the content from the web.