I spent 3 months scraping 6 million pages from an extremely old political web forum (I think their backend was written in perl and hasn't been updated since 2001 and the response times for these pages were extremely high) in an effort to learn NLP stuff with a working database. Its cool because you get to see how old Boomers digested huge events during 9/11 up until now and the slow decline of web forums as a medium. The pain is trying to parse the actual pages since the dev never learned how to liberally use div tags like we do now.