SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
Conversely, there are lots of folks who love ops type problems and are not well suited to building apps. So, it makes sense to put them on separate teams where they can each do the thing that they're best at, rather than fumble through the parts of the jobs they hate. But yes, at this point it's not really DevOps anymore, it's just ops.
Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.
The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.
It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.
You’re so right. This is the key, whether a team is doing devops or not.
One of my major contributions when I took on a CTO role was getting dev, ops, and support teams to talk about the pain points. There were so many easy wins to make things run better because the teams didn’t understand the difficulties the other teams were going through.
Some of the best meetings I setup were with head of the CX (Customer Experience).
The reason is SREs typically work on Infra/Product, but rarely see the pain customers endure outside of major incidents.
I've found that "ops" has become far too overabstracted and jargony. Their moat is endless "certifications". Their "experts" are the dogmatic personality types who try everything in their power to prevent their feet from ever touching the ground like it's lava. They lack the kind of dev or sysadmin experience where you have to stand under your own constructions like a Roman engineer.
Any formerly-known-as "full stack" dev who kept at it for the last decade and a half would run circles around anyone in ops. That's every senior dev working right now who doesn't like to manage people.
It used to be that the ones who went SRE just wanted a pay bump for a bullshit job of relatively light coding puzzles and parroting stuff in meetings, but I guess that well has dried up with "AI".
It turns out about as well as if you set the Ops team to shipping features.
Eg I’ve seen more than one full stack dev team build a synchronous RPC-over-message-bus framework because they can’t figure out how to make HTTP or actual RPC calls reliable. It is hands down the most poorly and overengineered thing I have ever seen on a service that takes single digit qps.
The ops teams fight that shit tooth and nail because if it blows up, they’re the ones who get called. And it is 100% too late to fix it by then. Ripping it out will take months, and the dev team will say they’ll do it, but they never will because the VP is leaning on them about some feature.
You either say no early, or you are forever on the hook for whatever abomination the intern has cooked up.
But if you have the budget and connections to hire a department of people with these skills and experience, and a problem interesting enough to keep them engaged in the day to day building, and a leader capable and charismatic enough to align them all into the business's goals... you should do that.
Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.
Your comparison might work better for a dev with no testing experience. Write the code and throw it over the wall. But I fail to see how an application engineer should need to worry about the intricacies of, say, setting up virtual private gateways.
That’s not to say you can’t have operations experts or ops-only roles. You can. But if the developers don’t participate in operations at all, they are doing themselves and the business a disservice.
It’s exactly like having a chef who doesn’t taste food, or a car mechanic that doesn’t drive. Some savants exist who can do good work even then, but they’re vanishingly rare.
In the early days of devops it was primarily focused on supporting developer productivity (CI/CD) but quickly morphed into sysadmin once cloud computing took off.
Some places do have really well built out platforms where logging, instrumentation, databases, deployment methodologies, IaC, etc have well-built happy paths that exclude a bunch of accidental complexity.
I think a lot of other places are either too small or grow too quickly with the wrong set of values and admit too much into their "platform" which had very little design to begin with.
Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.
conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.
To my understanding, this _is_ DevOps in the original sense of the word, and the version where they are a separate team is an offshoot bastardization.
At my last job, it was somewhere in-between where we had DevOps specialists who were regular software engineers who allocated the bulk of their time to building / running infra via Real Code (Pulumi vs. TF, minimal YAML wrangling etc) and built paths for app devs to self-service as much as they could. That, I felt, was a really excellent setup.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
Because they'll just offshore you
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
> The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
Sounds like a cultural issue. Google should look into that.
You didn't look around very hard then. I acknowledge that certain areas of the company can consistently use subsets of a role, so you may not have encountered it personally, but your view is definitely wrong about the role in the company as a whole.
Easier to shit on people and things with hot take then actually have a grounded and coherent argument or realise that different companies/teams have different constraints, problems and environments they operate as well as different solutions.
It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
Uh, I can literally see thousands of crashes happening all the time. You think there isn't a .01% chance of hardware going bad or hitting kernel bugs?
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
Other people's code? Not so much, I trust that a lot less.
Code of Hammurabi's Law 229: "If a carpenter has erected a poorly constructed house, so that the walls cave in and kill the homeowner, that carpenter shall be executed."
I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.
If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.
It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.
It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.
If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.
With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.
I think you’d be surprised if you look at the younger (post-2000-ish) companies. Specifically about this part:
> If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that.
You would be surprised how many workloads either are simple, or can be ham-fistedly shoved into a “simple” architecture.
Eg I’ve seen a lot of Postgres-on-k8s solutions that are basically just Postgres on bare metal with a bunch of useless abstractions. It writes to local disk, moving the pod to a new host is basically restoring from backup. Wrap it in some tool that ingests YAML and call it a day.
k8s is sort of the king of this. I hear people say they understand k8s and I can’t help but furrow my eyebrows. I don’t even pretend to, there are so many insanely complicated edge cases.
Eg if I’m sending UDP packets to a service and scale the deployment from 2 pods to 4 and the host for one of the new pods crashes and comes back up with a bad iptables rule, what are the range of potential behaviors k8s would show? What’s the difference between blocking say all traffic vs only traffic to one or two of the etcd instances? What if it’s blocking half the hosts in the cluster?
Storage has its own weird vagaries around single vs multi mounts and race conditions therein.
k8s is the king of “I don’t have to know that, the platform will do it”. I’m not sure I’ve ever heard anyone outside networking talk about whether a pod will saturate a NIC. Just chuck it in the cloud and let it figure itself out.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
SRE just came from the big guys calling their sysadmins SRE
This caused a lot of money to be poured into all kinds of projects, not all of which had a positive return on investment.
Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.
More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.
What the fuck do you think the DevOps movement was about?