As a reference the archive of all Reddit comments from October 2007 to May 2015 is around 1 terabyte uncompressed.
You could do exhaustive analysis on that dataset fully in memory.
You could do exhaustive analysis on that dataset fully in memory.
I only point this out to try and correct a common error I see. You're absolutely right that it is awesome that the entire data set can be analyzed in RAM!
What time of day the accounts most frequently comment. (I'd bet there is an interesting grouping of those that post while at work during the day, and those who post from home at night.
or what subreddits people comment in most during the day vs which /r/ they post to at night ;)
Sounds like you'd enjoy reading this guy's stuff: https://www.reddit.com/user/minimaxir/submitted/?sort=top
Looking at the thread from the release, I see no explanation of how he got the data, but I see several people commenting that they finally have a way to get comments beyond the 1000 per account: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
Practically speaking, on the other hand, reddit likely never took user content licensing into account to begin with - they never knew they'd get all of digg a couple years in, and 2006 was a different time besides.
The only way reddit would be able to release the data now would either be a specific clause (added from the start) allowing such in the TOS, or a general-purpose "we can do w/e we want with your data"; nobody thought to add the former (as noted), and of course the latter would have been roasted alive the moment it was noticed.
So, me thinks licensing issues.
Modmailing /r/reddit.com is the canonical way to reach the admins (for any reason), if you'd like more info.
The storage format of a dataset can make a big difference in memory usage.