Comment by brk | Hacker News Reader

Comment by brk | Hacker News Reader

Parent Full thread

Get your own ASN ;)

You are on the right path with your approach #1. Setup a failover server someplace and keep it sync'd at an interval that is the right balance between too-often and too-stale.

You could opt to get a small dedicated machine from a larger-scale true multi-homed hosting company. Run haproxy on this "small" machine, which serves primarily to redirect traffic to whichever server is currently deemed to be the "active". Basically this is a low-end DIY Akamai kind of solution. Your cost would be somewhere between #1 and #2. You would eliminate the DNS-lag and the issues you can't control (other servers and clients caching stale DNS data that you can't refresh), and you would get most of the benefits of an availability service like Akamai, without the $15KUSD monthly price tag.

There are other benefits to the haproxy solution as well, you can take either machine offline for maintenance with zero (theoretical ;) ) downtime, and if you ever get a traffic surge you can load balance between the two sites (which isn't a bad idea to do all the time if your syncing is up-to-date enough. Send every 10th connection to the fail over site, just to make sure it's always working as expected).

You still have a single point of failure with the haproxy though. i.e. if the Data Centre with the proxy is taken out (same probability as the prod site), then the service would stop.

Even if you install multiple haproxies you're either back to the DNS scenario, or a more expensive solution as you were suggesting.

If you are with a large-scale multihomed hosting company, then you might be able to manage with an IP address takeover between sites (ie. the passive haproxy takes over the IP of the master in an event of heartbeat failure). I've not seen this as an offering for the most part though (at smaller outfits anyway).

You still have a single point of failure with the haproxy though. i.e. if the Data Centre with the proxy is taken out (same probability as the prod site), then the service would stop.

That's why I said a true multi-homed datacenter. Reduces the chances of a single-point failure taking everything out. It is going to be somewhere in cost and resiliency between relying on DNS and "doing it right".

In all reality (and my own experiences) data-center outages to various "natural/random disasters" are far less common than outages due to just basic employee stupidity. While I was building up my org/website, I would spend more time worrying about resiliency against stupidity vs. plane crashes.

Well I guess my logic is that if you have each site either multi-homed, or at least on different networks (a given anyway), then you can have a LB at each site with the same net result - this is basically your vanilla active-active scenario.

In either case - Unless you go for a high-end solution, the catastrophic-scenario-DNS-juggling doesn't really go away.

As for the relative risk, I agree. The problem is the risk approaches companies use tend to make this kind of thing compulsory for them.

All the raw calcs I've seen rarely justify this kind of expense - the cost-benefit, against, say, a 1 day turnaround to configure a new site (two sites is usually a lot more than double the cost as there is all the other factors e.g. additional/routine testing that's introduced).

... but then companies will factor in less tangible risks, like reputational damage, etc, which can be a bit of black magic... Of course, if you're a bank, or a hospital this might be right, but I suspect in most cases it's off-base.

What happens if the 'small' server fails?

Depends on how risk-averse you are. You can generally get an SLA that would allow you to have a hot-back up of that server, or you take the risk that it won't wail until you can afford the "better" solution.

Doing "high availability" coupled with "low budget" is ALWAYS going to involve some compromises.

If you keep your DNS TTL's low, the worst-case scenario is if the small server fails, you're doing exactly what you would have done anyways: making manual or automated DNS changes.

When I investigated a low cost solution like this before, the issue was big companies like AOL not honoring TTL's less than 24 hours. Is this still the case?

If the client is keen to keep costs down... and is amenable to it... then you could give him an alternative URL to use in the case of a catastrophic failure (i.e. your plane scenario).

Then it's up to you if you use two haproxies for active-active or just maintain two active-passive sites.

Reply on news.ycombinator.com