Comment by conesus | Hacker News Reader

Comment by conesus | Hacker News Reader

NewsBlur's developer here. Just want to point out that this bug has bitten me back in 2.2, a few months ago. I need to be able to replicate my db in order to shard. But the biggest beef I have with this bug is that it doesn't become apparent until AFTER you spend 6 hours replicating (which results in poor load times for the primary db).

I would love it if I could choose which server to sync from. This option used to exist, but they removed it. But that would only solve the performance penalty of replication.

To their credit, the MongoDB folks have been stellar. I had a hardware failure a year ago and the CTO ssh'ed into my machines to figure out what was wrong. This time I'm having a bit more difficulty getting the problem fixed, but it's understandable as I have 100GB of highly variable data.

100GB takes 6 hours? That's ~4.7MB/sec - is that right? That seems, sorta like slow?

Not to sound rude, but what is so difficult about managing 100GB of data? You can fit that in RAM without much work.

The transfer speed is between data centers. West coast to east coast.

As for the 100GB, I have 12 task servers with 6-8 processes each reading 25-50 stories each every 4 seconds. I also have 18 app servers with 6-8 processes each reading 12 stories every second. Each story is on average 4KB. That's several MB a sec.

I just signed up for newsblur and paid $24. Love it.

Why didn't you just take a snapshot of the DB and start that up before adding it as a replica set member? I've had better luck doing that.

Because that server isn't in the cloud. It's on leased hardware, so no ability to snapshot. Plus, I like it when I can replicate my data without worry. I don't want to have to rely on block-level replication.

base698 isn't saying take a snapshot of the server, he's saying take a snapshot of the DB; find the directory it is stored in and `cp` that.

That's not a good snapshot unless you're running LVM or can afford to suspend writes and fsync + lock.

And again, this is yet another problem solved decades ago by RDBMSs.

And even earlier by Polaroid!

Armchair quarterbacking here of course but do you think it would be feasible/a good idea to try to switch to a less..troubled product at this point? I don't hear many horror stories about Cassandra for example and AIUI that's serving most of reddit.

Reply on news.ycombinator.com