Hacker News Reader

Yes, I done exactly this. it is actually 20GB uncompressed and takes 5-6h to parse. http://appraisalmind.com/articles/wikipedia-word-frequency-l...

That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.

Would you consider making it available to others?

Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.

archive.org will host datasets.

To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit...

To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.

This is why GOOG silently backed Wikipedia: to gain quality content regarding syntax and semantics. Other than give people a "good reason to surf" and hit "ads" [trackers] down the road.