> The problem is most engineers don’t want to do operations work.
There's your problem. You have people who build stuff without caring where and how it runs. Recipe for disaster.
There's your problem. You have people who build stuff without caring where and how it runs. Recipe for disaster.
There's a reason to have a team of people doing this "DevOps" work. Just like we have a team of people who do SRE. It creates a standard and a single point which all work flows through. Then you don't wander onto a new project only to realize they use $BESPOKE_DEPLOYMENT_METHOD because "it's what we used 6 months ago". Or worse, you don't have a developer playing with a massive, nuclear powered, foot gun like Terraform and accidentally destroying infrastructure.
Making DevOps/DevSecOps/$BUZZWORD the responsibility of developers is a cost-cutting measure not a responsibility measure.
If you're building a crud-app in a common framework with low volume, sure you can toss that over the wall.
In theory, devops shouldn't be responsible for maintaining the performance of code. The specification should say what it should run on, the devops guys set up a pipeline and manage that thing, and the developers are the ones taking heat for not hitting that goal. If devops guys are taking the heat for that it sounds more like cost-cutting measures flowing the other direction.
Yeah and this is a great solution (aside from the bus factor), maybe the person I responded to was really concerned less about knowledge and more about expertise (the word they did use in their was expert) At a company I worked with we called this T-shaped engineers. Deep in one thing, but broadly knowledgeable. Devs have to have knowledge of ops, but not "expertise" that is ultimately what ops is for, we may just ultimately be fighting over where the knowledge line is sufficient and what constitutes "expertise" :-). I for instance think terraform is not that much of a footgun and provides good rails to be used by developers.
And in some ways this is by design, I want to have some distance between dev and ops because it gives me the freedom to rearrange infrastructure transparently. I can move workloads between Lambda, ECS, and EC2 based on the observed performance characteristics without anyone being the wiser.
I think most modern AWS services are more the equivalent of an API or a microservice than they are a server, and you need to understand the limitations of the services you integrate with. If you're a cloud native company and don't have a mature platform engineering team, devs are going to have to know alot about AWS.
If the developer has all the information necessary to create an S3 bucket, lambda function, kinesis stream, etc does it make sense for them to offload 10 pieces of information to me, or for them to learn HCL and interact with it themselves, especially if it is something they do often. Especially if there's a central dev/ops team and they're the limiting factor. Devops taking over every infrastructure change for a broad team of devs is inefficient and expensive. It's also probably frustrating for devs that are aware of the above factors. Lots of devs I know would prefer to do it themselves.
And all of this is to mention that infrastructure and integrations with infrastructure are not static. Should I be reviewing all PR's to a system that touch the client code for an AWS service, because the dev team doesn't want to learn how that service works? Maybe, but in the end I don't know if this makes anyone happy.
Platform engineering is certainly the goal. Where ops creates a platform and dev just consumes that platform. But I don't know if it is realistic. Every system on rails is great until you try to take the rails off, and most devs I know hate rails :-) Fundamentally "devops" is all meant to solve human problems, not tech problems so it will have to be dynamic.
"I don't want to have to worry about what machine my app runs on" vs "kubernetes is to complex"
"Dependencies change to often" vs "I don't have time to maintain this thing I wrote myself"
"I just want the infrastructure to figure out what I need" vs "I want to be able to build whatever I want with a bespoke language/framework/database/architecture"
> it's a knock against the tech itself that it forces all this incidental complexity on you.
If I were to say that Kubernetes is the magic secret sauce that fixed all the incidental complexity I would get laughed out of the room. There is no magic secret sauce to the incidental complexity, the more we try to fix it the more we create (or the cheap-fast-good problem. This is probably easy if there are no cost limitations)
When I design a backend system, I need to think about how the front end developers are going to interact with it. My data storage characteristics and scaling. I need to know am I designing anything that’s hard to deploy. How will logging work and be aggregated. I have to be able to think about the entire system.
It’s not just “cost cutting” at a certain point in your career you are expected to know more than just “how to code”. I’m not saying learn AWS. But I would expect any senior developer to know about what their code runs on top of .
I think to push back is rightfully coming from the “ops” part. I consider “creating the CloudFormation/CDK/Terraform” code as part of “development” as part of coding.
If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Yes I knew AWS pretty well by the time I left and I needed to know it to be a good developer in that context and designed most of the processes around it. But I refused to do “operations” - ie “infrastructure babysitting”
There is a huge distinction between “I don’t think I should have to know how everything works” and “don’t call me in the middle of the night when something goes down “.
[1] I’m the first to admit that I left the “real world” once I started working in the cloud consulting department at $BigTech
> If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Sure you could argue a developer could, or even should create these things in theory. The problem is then when it goes down I have made two problems out of one. Now I have to manage the infrastructure of a system and what is running on it. Realistically, and even in my current job, it's actually several systems. Now when something breaks I have to pray I can fix it. Instead of allowing a team of infrastructure professionals to at least insure the hardware is working my 8 hour day turns into 14 or 16 very quickly the second one thing goes wrong.
Same shoes, but I have different perspectives. I can figure out where it's not working and if it's my/our team's area of responsibility then we go fix it.
Since we handle infrastructure (as code) and deployments in team, along with all the development, most of the problems are handled by us. Unless it's clear that it isn't, e.g. some API that we consume that keeps throwing 500, can't fix that.
Our operations is helped by 100s of automated tests and 1000s of metrics. I always thought that this is DevOps, but it sounds different from what most of the people here are alluding to.
They’re tools for the job, like your compiler and the language you use to program.
That’s like saying “I like to use Python and couldn’t care less for Java”. It’s fine to disagree with the team’s choice of the tools, but one needs to eventually commit to the choice, even it’s not your preferred one!
There’s an old adage that “if you’re writing clever code then you may not be clever enough to debug it”. It’s true! Operations requires deep understanding of the code running in production, the business rules and the customer. The person who will operate your code will eventually be smart enough to develop it entirely too, eventually cutting out the “dev”. I’ve personally seen this happen over and over.
My background is large multinationals so my view here is a bit bias but i don't think cost cutting is the driver.
Large orgs get large change management processes and procedures. Over time, these change management teams become overwhelming behemoths with minds of their own.
I think "DevOps" was designed as a way to "bypass" the bureaucracy?
"We just use this CI/CD pipeline and no need to sit on a 3 hour change management review call..."
It becomes just robots pushing around paper for compliance.
If you don't want to think about AWS S3, GitHub Actions/Jenkins, Terraform, etc, .... then we need to work together. All those tools and services exist because all the software developers are sitting in their sandbox, and don't want to come out and play. The systems and tools that we run your code with... suck. A lot. We need programmers to make the systems better. We (in Ops) are a little busy with trying to just figure out how to run your apps without them falling down. We don't have a lot of time to reinvent the state of the art of computer systems.
For example, we need a distributed operating system. Not some fucked-up kludge of a monolith of microservices overseen by a company that has more engineers than brains... but an honest to god, stable-ABI, simple, composeable, stable, general operating system. We need Linux to come out of the box, ready to run distributed applications, in a way that doesn't require a PhD. Once we have that, then you - yes, you, the developer! - will be able to make applications that automatically scale so easily that we will never need to utter the phrase "container" ever again. You will rarely ever need us again, because the system will just be so simple, so general, that anybody who can use the terminal can build and deploy applications without ever learning anything outside of your programming framework.
But we need you to make that distributed operating system. Until you do, we will just have more stupid kludges, more bizarre unnecessary complexity, in the futile attempt to constrain all the crazy shit we want to do with technology, while trying to run your apps for you. Please, I'm begging you - put me out of a job.
Most developers have to do Frontend, Backend, and Ops. These have wildly different mindsets and feedback loops and not enough time exists. Don't hate the player, hate the game. The orgs are fucked, not the workers.
There are only so many hours in the day, it’s hard enough to stay on top of my core skill set.
Unless you're arguing that engineers are unproductive members of a company, I suppose.
Software devs who close their mind to the lower parts of the stack give me the opportunity to learn and do their jobs as well, becoming a more valuable employee as a result.
That’s iyho. The management of a bus line company doesn’t see it that way ;)
The word "scam" is overused. It's a scam if it was a bait-and-switch. It's not a scam if it was advertised this way up front. In the latter case it's just a job whose description you don't like.
It is not better for the department members or the software itself. It is only good for the bottom line. Yes, if you know it all you'll be valued higher - but the company is still reaping the majority of that benefit from you and not rewarding you with 3 salaries. You should not perform all the roles because it's economically idiotic to do so. You could contract out all 3 services and make a shit ton more money than you would performing them in a salaried position.
That is a scam.
If you think that is a scam, I don't know what to tell you.
I'm not sure why this happens but it does and I don't like it (as somebody who's very M-shaped).
Letting them wade into others responsibilities often introduces a lot of that danger.
System administration is an entire discipline with a history and way of thinking and approaching problems at a systems level. Software engineering is the same, but with an approach that often emphasizes handling problems in the application.
Letting solutions be put into applications that are best handled at the system level (where system isn't just an OS, but might be a complex network of them), or put into the system when it could easily be handled by the application, caused inefficiency and problems.
The solution is to either hire only very accomplished people that can do both and make the right decision of where to put solutions, or hire people that can do one or the other and put them together with a few (maybe one) person that has a good grasp of both and can make executive decisions when the right time and place is for specific solutions (an architect).
The latter is easier to hire for.
I'm also not convinced that the average junior or mid level application dev needs to know where or how their stuff beyond some high level concepts (i.e. our app uses auto scaling).
I've done some, and it feels like reading an arcane text in another language. I've seen the thing I want before, so I know there exists a magical YAML incantation that brings it about but I have no idea how to even start looking for it. It devolves to an extremely slow feedback loop of "did that work? No."
I've never found anything that explained how I'm supposed to create a systematic approach to creating anything I need.
"That didn't work, and the traceback (if I read it) isn't very helpful. Guess I'll try something else I found on SO."
You learn ops by doing it, preferably in an environment where when you inevitably do it wrong, it doesn't break prod.
Starting in development requires a laptop. Starting with operations requires a datacenter worth of hardware and applications.
In the old days, developers would also configure the design of the system it ran on.
"Alright, we're going to use some fast CPU front end nodes with smaller, faster disks, and then have a load balancer, our database server is going to have the most RAM, now let's set these resources up on a CDN to serve faster, this memacache server is really going to lighten the load..."
I think the idea was to use tools that automatically did all this work for you, but it just ended up creating another tier of people that configure this side of it.
Though mind you, the article touches - but doesn't go into - the real reason things go shit. It's not ops, it's not devops, because your knowledge silo can too, and in my experience (that's longer than the writers, lol) _will be_ pressured.
If the estimates are consistently too low (because honest ones would render many - maybe most - projects non-viable), the requirements are often bullshit (because otherwise the lies in the estimate would be too obvious, and everyone needs to agree on the lie) and and rewards are not the ones the jobs claims to be (in every workplace, your job is to improve your resume. That's all. That's the only thing that, materially, matters for your career. I can care about software I write, but if anything I end up punished for it) - then no amount of renaming things and shifting org charts around or motivational reading for managers is going to fix that. Nor will a Cloud or latest orchestration software.
You don't want a doctor who brags that she can do heart surgery, perform talk therapy and treat your skin rash. You want someone how knows a lot about your specific problem because she's prioritised it over other things.
Perhaps most of our organizations are using the wrong tools and that the way we divide responsibilities is the problem.
Imagine if:
1.) instead of general purpose languages being used by product developers we had domain-specific languages and tooling that were agnostic to things like deployment, memory-management, data storage
2.) computer engineers built the domain-specific languages and tooling and focused on things like deployment, memory-management and data storage.
DevOps is generally solving for general purpose development while developers are generally solving domain-specific problems with general purpose tools. This seems to be a large part of the disconnect!
I’m currently working on a small team of computer engineers who support a custom Turing-complete DSL for an “operations” team that both programs and directly interfaces with clients.
It doesn’t seem that different from what Apple or Microsoft do, which is write custom languages and tools for developing on their platforms, albeit specifically for general purpose programming.
Why would someone who wants to build stuff not care about where and how it runs? Operational requirements are the same as any other requirements.
Spot on, but not only that, also decisions made based on number of hands in a room.