Try this. Tell them there are three models that regularly achieved five 9's with low latency for over a decade each. Each with clustered or HA configuration over dedicated lines. List them as such:

1. IBM mainframes.

2. HP NonStop systems.

3. OpenVMS clusters on SMP machines.

Give them the prices of systems and rare labor for each. Then, show them the price of a three to four 9's solution with Linux that has three to four less 0's. Mention how many extra bonuses, err business investments, they could make with such savings. At this point, they might treat you like an idiot for not buying the cheaper route. Maybe even enthusiastic about your business savvy approach. ;)

That's interesting and probably more in line with what I need to do. It wouldn't be difficult for me to bring the entire stack online in another datacenter, but if all of your customer data resides on (for simplicity) a master SQL node in Linode-Dallas and that datacenter goes offline, how can you bring that backup node online when you can no longer access the Dallas machines?

Are you suggesting to have a slave ready to become master in the 2nd datacenter? If that's the case, my main area of uncertainty is: if you promote a slave to master in a 2nd datacenter and that new master accepts writes, you then have a brutal problem of propagating writes to the old master when he comes back online.

The whole thing may exceed my capability right now but I think having a disaster recovery is far better than a mirror now that I think through it.

I'm curious about the stuff that's hard to replicate. Don't most of them store their stuff in files? Could those users perhaps use a distributed filesystem for replication? Clustered filesystems were used in supercomputers, are used by Google, and similar methods go back to VMS systems on VAXen.

I'm not saying something off-the-shelf will work or out of the box. Just seems like there would be a clear path for admins or developers of such software to integrate it with... something [1]... that did that.

[1] Example: http://sector.sourceforge.net/

That's a proven approach going back decades. A number of banks do it, too, because it often works well enough in practice where data isn't lost. A trick on that is having something on each end that sort of checksums the data(base) for certain time periods to see if anything was lost. Then, resends that stuff. My use of that was inspired by the transport layer of the Tsunami file-transfer protocol with the implementation very similar, too. :)

I think it is a good idea. It is a bit more complex than using Unison, but it should be more generalisable.

I think the real key to creating a simple and low cost reliable system is to have this in mind when you design your application. I could have used a database just as easily as a flat file approach, but this would have made things much more complicated to replicate between the nodes.