Stack Overflow running short on space
nickcraver.com
nickcraver.com
"...the server goes to 288GB (16GB x 18 DIMMs)…so why not? For less than $3,000 we can take this server from 6x16GB to 18x16GB and just not worry about memory for the life of the server. This also has the advantage of balancing all 3 memory channels on both processors, but that’s secondary. Do we feel silly putting that much memory in a single server? Yes, we do…but it’s so cheap compared to say a single SQL Server license that it seems silly not to do it."
Clearly - if you are paying 10s of thousands of dollars for a database server License, it makes sense to fully utilize the ones you've purchased.
Also, in my experience, databases these days pretty much have to stay in memory in order to be performant whatsoever. I think the rule I heard from a FaceBook DevOps manager who was interviewing with us was "If you touch it once a day, it goes into SSDs, if you touch it once an hour, it goes into memory." - of course, at a certain size, you _have_ to scale horizontally with those in-memory databases as well.
I suspect this is the reason they're using a single server. It would cost a few more tens of thousands of dollars to install another one, versus the zero overhead of just upgrading the existing one.
As an OSS user, I often take for granted how easily I can go "Oh, the database reads are too slow? Let's throw another Postgres slave at it and see how that goes." After all, that expense is usually minimal.
I think that what's happening here, really, is the emergence of SSDs as a server platform standard. SSDs drastically revise the algo-economics of data storage.
From the mid-90s until now, demand for structured data storage has surged, apparently exponentially (though realistically, such a function would eventually be sigmoidal). So dramatic has this surge been that it overwhelmed the somewhat linear rate of I/O performance improvement for disk drives. Meanwhile, Moore's Law has more or less held and so in-memory architectures have become much more popular, starting with memcache.
But SSDs change the equation, because they too can track Moore's Law. So they can take mature disk-bound technologies like RDBMSes and breathe new life into them. And that's exactly what's happening.
At the turn of the millenium, if you were given StackOverflow levels of traffic and everything from 2011 technology except for SSDs, you'd have to spend dozens of times as much to approach the same level of performance.
Yes, that is the problem isn't it?
The Win stack is a trivial cost in their business compared to the revenue potential of StackExchange.
The biggest cost in their business is labor, not licenses. That will always be the case. License costs will continue to shrink toward irrelevant as a percentage of their sales over time.
Given their wild success, there's no great argument to be made against what they've done: quite simply, it has worked, and worked very well. That is all that will ever matter, regardless of what argument is thrown around.
It's certainly a factor worth weighing when you consider what technology you'll build on top of.
In no place have I "argued against what they've done" or implied that the company is a failure. Yet money IS a part of scaling, and managing expenses IS a part of business. I'm happy to take your word that Stack Exchange makes so much money that it cannot ever matter how much their licenses cost. But in the context of evangelism for others to make the same decision, I have to observe that the kind of reasoning casually mentioned in the article implies something that would be negative for many other businesses.
If you are offended by that sort of discussion of reality then you have too thin a skin (or a conflict of interests)
We (Defensio) store about 300gb of data, and most if it is purged after 60 days. We're quite far from being 274th largest website in the world as well, I assume.
It's just very interesting to see how such a huge website can use so little storage.
... I suppose for spam, the problem is that as a side-effect of trying to fool Bayesian filters, there's a lot of incompressible gibberish.
One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression.
In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on.
Compression would help a lot, if it was available (some brief Googling suggests it isn't).
(edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique.
And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better.
Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins.
Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.
MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock.
I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...
One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs.
I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock.
The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.
These guys should open their wallets and get more than a little bit of SSD, prob PCIe, plus max out on RAM. 96GB is what most would start at now in a larger HP server.
Also, with FusionIO you are putting all of your eggs in one basket - drive wise. If that card dies you are done. In short we don't _need_ the FusionIO so why would we put out that kind of money for it?
Also we went to 96GB about a year or so ago when we setup those boxes it was a reasonable amount of memory to put in now - we are maxing the thing out at 288GB now.
While you mentioned SAN storage being really expensive (and the price changes depending who you are), you really can't beat the caching modules. I've got a Netapp with 512GB of read cache, and you can buy them in smaller sizes.
Have you guys noticed a degradation of SSD performance? I've never used SSDs in a machine before, and I know Anandtech always writes about the performance degrading after the drive gets mostly full. I just wondered if you've run across the problem in production.
Also you can mirror them.
Scaling a static Q/A site with a few widgets that require user info/counters should be table stakes.
The thing is, they don't need to do that. Moore's Law, here represented as SSDs and RAM, is keeping ahead of their traffic growth.
I think they've successfully shown how far you can take it. Nevertheless, I agree with you that scaling this horizontally makes a lot of sense.
Most people's traffic/data hasn't grown nearly as fast as Moore's law. Scaling up makes more sense than ever. It may not be hip but it's the right choice for 99% of cases.
Bottlenecks are not magically confined to only affect scale-up designs. And if you look at the design of scale-out systems, they usually surrender some desirable quality such as easy joins, isolation, atomicity etc.
Case in point, here's a 16TB SSD with 560 MB/s peformance: http://www.engadget.com/2012/01/09/ocz-goes-ssd-crazy-at-ces...
And that's one of the least revolutionary implications of memristor technology.
Memristors look extremely likely to be able to provide non-volatile storage with latency, access speeds, and density higher than existing DRAM and flash. They also look like they can be used to make FPGA-like devices which can approach the speed, power efficiency, and logic density of custom designed ASICs but with extremely fast reconfiguration speed. And those seem to be the less fascinating aspects of memristor technology.
I guess the static content is CDN but all dynamic is coming from one machine?
Oh wait, nevermind (10 Dell R610 IIS web servers)
http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
I think they're using SQL Server's inbuilt text search engine (edit: no, see below).
Summary: Slight benefit to read performance and greater resiliency in the case of drive failure with Raid 10.
If you run RAID 5 or 6 and a disk fails, suddenly every read operation becomes several times slower because the missing data must be computed from the parity and remaining data channels. If your normal day-to-day load on the storage is too high you are screwed until you rebuild your missing disk.
Recovery will drop read/write performance, depending on how fast you do it, but nowhere near as bad as RAID-5/6.
RAID5 is notoriously bad for write performance, just as RAID6 is. This is a classic writeup on why you should avoid RAID5 at all costs for a scenario like this: http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt
Crucial M4 256 GB (4KB Random Write: Up to 50,000 IOPS)
or
Plextor M3 Series PX-256M3 256GB (4KB Random Write: Up to 65,000 IOPS)