Google has also released a full-web 5-gram corpus that contains "counts for all 1,176,470,663 five-word sequences that appear at least 40 times" Its fun to try training on that.
http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...
http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...
One of the painful things about commercial NLP work is lack of good datasets without restrictions. There are some treasures like WordNet (which is a lexical db). For tagged data I found the American National Corpus but the excerpt they release without restrictions is too little data and it contains only a few styles of writing.
I had to spend a lot of time gathering, fleecing, and marking up my own data to make After the Deadline.
Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit...
To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.