Crawlers and anti-crawler wasted too much resource, data shared in open database is a great idea.
BTW, if someone can maintain all data listed here, that will be really helpful. https://github.com/awesomedata/awesome-public-datasets
Great dataset covering government, economics. Hope that more open data can be added to this database for research.