Hacker News Reader

Comment by colinhowe | Hacker News Reader

We have a dataset much larger than RAM (600gb vs 60gb RAM). It's the working set that really matters. Accessing data outside the working set can be slow - unless you have SSDs :)

Regarding consistent/durable: during the past four years, we've not had any problems on this front that weren't caused by us in. We've had an issue that was a misconfiguration on our part where we allowed writes to a server because we pulled it out of a replica set. We also ran out of space on the logging volume once and that caused downtime - but, we didn't have log rotation or anything setup.

In general, we've found the failover very reliable and new primaries have come online without any problems.

That said, schemaless is both a blessing and a curse. Now that Postgres and MySQL have online alter built-in I'd possibly choose one of them if we were starting everything again.

View on HN

Well, SSD's aren't nearly as fast as RAM, even today.

I think we are in agreement about the working data vs all data. I am saying that in most applications your working dataset is much smaller than your total data set. So why pay for hardware capable of holding your entire dataset in RAM when you don't need it?

I am surprised you are able to do this with Mongo. Last I checked, it simply did not handle this case, and started failing miserably if it was not able to fit all data into RAM.

Re: durability: I am not talking about the server going down, coming back up. I am talking about whether there is an fsync() when writing data. Set up a test case where you are writing data very rapidly to MongoDB, then pull the plug on the box it's running on. It'll come back up, but the data it told you it just wrote won't be there because it didn't checkpoint. Did you check that all your writes succeeded when you had node failures in your cluster? Most applications don't have the machinery to do this because generally the state necessary to check this is stored in the database, yet it's the database you are testing. The only way to test this is to also write logs (also atomically), and then verify DB data against logs. Or, just use a database that guarantees durability.

Re: consistency: MongoDB doesn't support transactions [1]. That's enough to exclude it from a large number of applications. Anything to do with money, for example, is out since you really don't want double spending to be a thing.

[1] http://docs.mongodb.org/manual/core/write-operations-atomici...

Fair points - we don't allow failed servers back into replica sets and rely on writing to multiple nodes instead of the disk as source of truth.

This may not suit everyone and absolutely does not suit financial transactions. You can bend Mongo to do it using additional collections and money movement logs... but, why bother when it's simpler to use MySQL/Postgres?

Reply on news.ycombinator.com