Comment by solatic | Hacker News Reader

Comment by solatic | Hacker News Reader

Parent Full thread

Ayyy, even the author gets it wrong.

DevOps is not about letting developers own services in production. This is the common misunderstanding that has lead to DevOps teams be platform teams. DevOps is about helping separate Development and Operations teams work better together so that the organization, which needs both, can benefit from their harmony.

The simple fact of the matter is that the modern stack is too big. It's not possible to expect to be able to hire someone who can walk in and do their own product management + write frontend code + backend code + host the frontend on a CDN + own the backend on Kubernetes + handle all the security, monitoring, uptime requirements around the clock. Maybe over time people will expand their skillsets, but right on their first day? No way. Not a reasonable expectation.

Organizations need to allow people to specialize. This is the only way anybody ever truly gets productive, they learn some part of the stack, learn in deeply, handle that part of the stack for the company, and communicate. True DevOps isn't pipeline engineering / automation engineering, it's corporate communication engineering, so that the concerns of people later in the value stream (ops, security) can be handled as early as possible (e.g. development/staging environments) to improve overall throughput.

You still need dedicated ops engineers! We just call them SRE now. You still need dedicated security engineers! We still call them security engineers.

Heroku came up with the idea of buildpacks [01] the idea has later been picked up by Cloud Foundry [02]. Using buildpacks (as an alternative to dockerfiles/images) mitigates this problem. A lot of best practices are build into the buildpacks.

Of course this does not work for everything. But for most cases this is just good enough.

[01] https://buildpacks.io/ [02] https://www.cloudfoundry.org/

Following this approach it becomes harder for the developer to do things wrong. This is one benefit why Cloud Foundry is used by a couple of big enterprises.

Letting/making devs own the services they deploy/develop is not the full picture, nor is it a real goal, but still it sums up the problem and solution space concisely.

Yes, specialization is a must for productivity, but that's a given already in the old model. What the DevOps effort tries to cut through is the trust barrier between the two. Sure, it's communications engineering too, but it's easy to weasel out of any real change with that phrasing.

> Letting/making devs own the services they deploy/develop is not the full picture, nor is it a real goal, but still it sums up the problem and solution space concisely.

What I'm trying to explain is, that I don't believe that this is a realistic expectation, because owning a service in production includes aspects of the stack that are not related to development, like networking, storage, and various non-dependency security-related concerns. When you task a platform team with building an easy-to-use abstraction for developers, those abstractions will inevitably leak. When they leak, one of two things happens: either the developers page the platform engineers, in which case you're back to shared ownership and the need to coordinate the concerns and responsibilities of multiple teams, or the developers are expected to fix the leak themselves, in which case, most of the time, the developers will find themselves out of their depth, most likely doing something inefficient at best or flat out wrong at worst.

I agree with the concern regarding leaky abstractions, but ultimately the developers should still own their services and be on-call for it, e.g.

- On-call dev deploys a buggy change that messes up some transactional data. Product owner passes the complaints received by customer service. On-call dev rolls back the deployment, writes a script to fix the bad data, and then prepares a post-mortem.

- On-call dev and on-call dba are paged because the database for the service is slow. On-call dev checks the service metrics and logs, and confirms that this is not caused by a deployment nor abnormal traffic. On-call dba checks the database metrics and logs, and finds some runaway queries due to silly optimizer. On-call dba kills the runaway queries, and proposes to add an index hint. On-call dev agrees, works on it, and asks on-call dba to review the change.

- On-call sec is notified because of a new rce vulnerability. On-call sec comes up with the steps to identify and remedy the issue. All on-call devs are notified to follow the steps, to ensure that their services are not impacted.

That's why ideally you have someone in your team who is trusted by the platform team to do what needs to be done to unblock your main goals/tasks. The platform team of course can help (or hinder), but the ability to do your own stack helps uncouple problems, helps teams progress.

Ideally there's an equilibrium. Some teams will inevitably be outliers, but the majority will benefit from the platform.

There's always a degree of sharing/coordination/communication. After all the teams are part of the same company. (Eg. when it comes to data access, API stability, it's a must to coordinate. Though famously Amazon "simply" solved this by a companywide mandate, requiring everything to have an API, and there's probably some story about their stability too.)

So, in any case the coupling should be loose. Basically an almost wait-free (but at least lock-free) algorithm.

Exactly, this is a huge problem.

A lot of developers are exactly that, developers. Sure if you have somebody who is very motivated he might know a thing or two about DevOps etc. but hiring becomes almost impossible if you expect a candidate to be a good developer + a good knowledge of devops.

It's just too much to handle for one person.

Reply on news.ycombinator.com