Hacker News Reader

Comment by joevandyk | Hacker News Reader

One of the problems with using multiple providers is you can't use any of the specific features of a provider.

For example, I really like AWS's security groups and ELBs. Those serve as my firewall and my load balancer and SSL terminator.

Replicating the application to another service means configuring and testing all that on my own.

If I use heroku and use their logging system, then replicating it to another provider means I need to be an rsyslog expert.

I don't really want to be an expert on rsyslog, postgresql configuration, floating IPs for HA LB, the best IO scheduler for file systems, etc. As someone who is in charge of all the sysadmin duties, and is solely responsible for writing all the business and db logic for several e-commerce sites, I want to spend my time on writing code. Not fucking around with figuring out the syntax for iptables.

View on HN

Ultimately this is why it would be nice for the code/configuration to be independent of the operations. I agree it is unreasonable to expect a random developer to build AND MAINTAIN the entire stack for every project. PaaS makes a lot of sense, especially as a starting place.

The solution is either to have a PaaS provider who ruthlessly eliminates single points of failure (there isn't one, currently), or use some standardized software system which can be operated by multiple independent operators with nothing shared. Unfortunately, the only vendor-independent infrastructure is the physical server, various forms of VPS, etc. -- it's all at the IaaS level. As far as I know there's no PaaS type thing with a common interface which arbitrary providers can operate, with some kind of marketplace for users to pick operators independently from the technology.

The solution is either to have a PaaS provider who ruthlessly eliminates single points of failure (there isn't one, currently)

This isn't possible by definition, right? The PaaS provider itself becomes the single point of failure.

Rings a bit like a "Who created God?" argument. "What single entity can I use to defend against failures by a single entity?"

Theoretically, sure, but it's an engineering and economic thing.

You can mitigate specific risks, and you try to prioritize those based on cost, frequency, and severity. If there were a great redundant provider with good authentication on accounts, a strong balance sheet and business, and sane policies on managing accounts, you would be fairly safe using just that provider. After all, you could always get a court order to cease providing services, yourselves, like if you do something some troll has patented. It kind of depends on your application, too -- if I were doing a wikileaks, a bitcoin exchange or torrent site or some other legally at risk business, I'd want country-level separation across multiple providers, at least as a cold backup. Casual game for facebook or mobile, not really much of a concern.

That's exactly why you don't want to use those features, for what it's worth.

That means that I need to know about (and program chef to do):

logging

security hardening

firewalls

backups (both point in time and complete backups, both for database and for any other artifacts like image uploads)

testing backup recovery (both the point in time and complete backups)

replication

SSL termination

postgresql configuration

monitoring (security, system stats, application availability, individual process running)

performance analysis (both at application and system level)

method for deploying updates (to the application and to the system)

and the list goes on... you could easily make a career out of focusing on postgresql configuration, for example. these aren't simple things to do.

Different providers have different limitations on what you can do here. For example, AWS doesn't support multicast, which limits your options for doing High Availability.

I think before you put an app on the public Internet for people to consume, you should be comfortable with everything you listed plus more. You might not be doing it for the app at hand, but you should know it at least. Especially if you're doing a solo founder thing, you might be having to implement all of these things before you can afford ops. If you're deploying services without some of the knowledge in your list, too, you're setting yourself up for trouble.

It's not an insurmountable barrier, for sure. Knowing how to operate your own code is a great feeling.

You do realize that it's a 6-figure investment to hire an engineer comfortable with all of those things plus more to deploy a multihomed setup?

A) My comment was very specifically focused on you, acting as a solo developer/founder, not a person you'd hire.

B) A multihomed setup is not a black box of mystery, and is nowhere near as expensive as people are saying from the hip.

> My comment was very specifically focused on you, acting as a solo developer/founder, not a person you'd hire.

Then "you," having acquired years of experience in designing and implementing distributed services, are making at least a 6-figure investment in opportunity cost.

Know about? Program Chef to do? How about: be on call 24/7 to handle random problems in? The best, most competent ops teams can all tell you horror stories about black swan outages traceable to some of the least likely components in the stack you just outlined.

Almost all of the things you outlined are things that have in the past blown up in real world deployments, often at cloud hosting providers where you didn't know about it it because a well-paid, well-trained ops team hid the drama for you.

So yeah, definitely factor that in to the cost of hosting at a cloud provider. You're absolutely right.

Reply on news.ycombinator.com