We (Defensio) store about 300gb of data, and most if it is purged after 60 days. We're quite far from being 274th largest website in the world as well, I assume.
It's just very interesting to see how such a huge website can use so little storage.
... I suppose for spam, the problem is that as a side-effect of trying to fool Bayesian filters, there's a lot of incompressible gibberish.
One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression.
In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on.
Compression would help a lot, if it was available (some brief Googling suggests it isn't).
(edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique.
And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better.
Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins.
Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.
MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock.
I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...
One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs.
I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock.
The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.