Hacker News Reader

Comment by danieltillett | Hacker News Reader

I think the lesson here is don't rely on one supplier. I have my tiny infrastructure spread over three different suppliers in different geographical locations. Plan for the worst and hope for the best.

Edit. This is in no way a criticism of linode. The worst outcome is if we all end up with one monopoly supplier. I have deliberately avoided using the big player in this space as I want support diversity. This makes my job harder, but it is better for us all if we don't put all our eggs in the one basket.

View on HN

Have you done a write-up of how you've managed to keep a real-time sync of assets/database? Or are you hosting static websites?

This is what I'd like to know too. It's easy to suggest mirroring your infrastructure on multiple providers but in practice it's very difficult. Multi-master SQL in particular is non-trivial and that's just one piece of the puzzle. We have MongoDBs, CouchDBs, Redis, and Elasticsearch -- each with varying difficulty and costs to cluster.

You're definitely not mirroring your infrastructure any time soon :). Not to say that it's impossible because I don't believe in impossible, but that's quite a stack to replicate.

We have much the same stack at Reverb.com and I focus more on creating a disaster recovery plan than tackling a mirror of our infra. I'm focusing on bringing up our stack in another region of AWS if we would ever need to rather than mirroring everything.

Sometimes a mitigation plan is far more useful than building in excessively complicated redundancy.

That's interesting and probably more in line with what I need to do. It wouldn't be difficult for me to bring the entire stack online in another datacenter, but if all of your customer data resides on (for simplicity) a master SQL node in Linode-Dallas and that datacenter goes offline, how can you bring that backup node online when you can no longer access the Dallas machines?

Are you suggesting to have a slave ready to become master in the 2nd datacenter? If that's the case, my main area of uncertainty is: if you promote a slave to master in a 2nd datacenter and that new master accepts writes, you then have a brutal problem of propagating writes to the old master when he comes back online.

The whole thing may exceed my capability right now but I think having a disaster recovery is far better than a mirror now that I think through it.

I'm curious about the stuff that's hard to replicate. Don't most of them store their stuff in files? Could those users perhaps use a distributed filesystem for replication? Clustered filesystems were used in supercomputers, are used by Google, and similar methods go back to VMS systems on VAXen.

I'm not saying something off-the-shelf will work or out of the box. Just seems like there would be a clear path for admins or developers of such software to integrate it with... something [1]... that did that.

[1] Example: http://sector.sourceforge.net/

Nobody said anything about multimaster. You keep a hot standby in a different availability zone and stream your transactions to it. Failover when needed.

Unless you're a bank, you run asynchronous with watchdog scripts and accept the very very small risk that you lose a few transactions within the ~100ms window of async latency.

It's all risk analysis. For most of us, hours of downtime is riskier than losing a tiny sliver of data for a small number of customers.

That's a proven approach going back decades. A number of banks do it, too, because it often works well enough in practice where data isn't lost. A trick on that is having something on each end that sort of checksums the data(base) for certain time periods to see if anything was lost. Then, resends that stuff. My use of that was inspired by the transport layer of the Tsunami file-transfer protocol with the implementation very similar, too. :)

I haven’t done a write up, but basically I just use unison [1] with a hub and spoke model. It is pretty simple, but it works well for my use case which is all file based.

My system is dynamic, but once a user is allocated to a node it is easy to replicate their data over the network. The key was I designed the system with easy replication in mind - flat files rather than databases, etc.

1. https://www.cis.upenn.edu/~bcpierce/unison/

Then what's your take on my suggestion?

https://news.ycombinator.com/item?id=10822904

EDIT to add: Good work on Unison btw.

I think it is a good idea. It is a bit more complex than using Unison, but it should be more generalisable.

I think the real key to creating a simple and low cost reliable system is to have this in mind when you design your application. I could have used a database just as easily as a flat file approach, but this would have made things much more complicated to replicate between the nodes.

I appreciate the feedback. This...

"I think the real key to creating a simple and low cost reliable system is to have this in mind when you design your application. I could have used a database just as easily as a flat file approach, but this would have made things much more complicated to replicate between the nodes."

...is a great point. It echos the comments and thinking of Bernstein in his paper on lessons learned from Qmail.

http://cr.yp.to/qmail/qmailsec-20071101.pdf

Particularly, section 4.5 "Reusing the filesystem."

This isn't necessarily directed at you, but I frequently see products which have no disaster recovery because they can't get something that seems good enough, contrary to any economical reasoning about what "good enough" is.

It doesn't always have to be complicated. Chances are It'll probably feel more like grunt work than rocket science.

Look up Disaster Recovery Metrics. You set an RTO (recovery time objective -- how long it takes to recover) and RPO (how much history is lost when recovering) based on what's needed for your project and feasible to implement.

Feasibility of disaster recovery strategy = Cost to implement strategy + Cost of losing data back to the strategy RPO + Cost of losing business during strategy RTO < Cost to business of riding out probable disaster

Pick the fastest/cheapest feasible strategy, implement it. Then once your ass is covered, pick the one with the greatest expected value and implement that. If you wait until you can design and are given resources to implement a 5-nines available strategy with sub-milisecond RTO and RPO (which seems to be a popular tendency of developers), disaster will strike before it happens.

Great advice, thanks. I am one of those developers who chases after "a 5-nines available strategy with sub-milisecond RTO and RPO" - because that's what the suits want, and I get treated like an idiot in meetings where I say it's not possible/not cost-effective.

And, y'know, because it would be really cool if I managed to ;-)

Try this. Tell them there are three models that regularly achieved five 9's with low latency for over a decade each. Each with clustered or HA configuration over dedicated lines. List them as such:

1. IBM mainframes.

2. HP NonStop systems.

3. OpenVMS clusters on SMP machines.

Give them the prices of systems and rare labor for each. Then, show them the price of a three to four 9's solution with Linux that has three to four less 0's. Mention how many extra bonuses, err business investments, they could make with such savings. At this point, they might treat you like an idiot for not buying the cheaper route. Maybe even enthusiastic about your business savvy approach. ;)

Reply on news.ycombinator.com