I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis?
That said, this is SO not big data. It is small data, but potentially interesting analysis.
That said, this is SO not big data. It is small data, but potentially interesting analysis.
I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of information theory or other non-bit-based complexity measures.
From this point of view, literature may not be large (in terms of the number of bytes), but in terms of information content, it is incredibly dense. A large portion of human knowledge is written somewhere. From this point of view, it is in fact big data.
http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
Here in his famous tour de force Claude Shannon lays out the way we can estimate the amount of actual information in an act of communication (e.g. a literary work) and relates it to system entropy.
To your point, !TB of all 1's compresses to a just few bits of actual information.
But I suspect you are not actually speaking about the information contained in a literary work. You are probably thinking: how much expansive commentary and explanation could a literary work spawn? That is another question. The answer is always: unbounded.
But after initial perturbations die out, the growth rate will likely look something like O(log n) or less.
"You have a terabyte of data? For $30,000 you can put that in main memory. If your dataset fits in main memory, it's not big data." Source: Michael Stonebraker, January 2015.