2023 DevOps Is Terrible

thats how it _should_ be.

The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies.

We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.

Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it.

In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..

god do I hate the "pet" analogy

yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them

Depends on the FAANG, Google famously throws devs at ops problems.

Ironically they realised this wasn't a good idea in exclusivity, so now they hire two disciplines: SRE-SWE and SRE-SYS.

SRE-SYS is as you would expect, pretty standard sysadmin knowledge from 2005 (IE; not helpdesk sysadmin, some scripting knowledge, heavy OS knowledge, heavy debugging knowledge).

SRE-SWE is much more programmer heavy with a light touch on OS fundamentals (based on what I consider important for OS knowledge FWIW).

depends on the company and department.

the downsides of SRE is that you tend to be super hyper specialised in something thats only really of any sense to that department inside a company.

Moreover, you're dealing with real steel machines, and amounts of data that are prohibitorily expensive to just start again with. (the joy of AWS is that most of the times you can delete it all and spin it up again, often side by side)

also, some SREs are really specialised software devs, others like me are sysadmins who'd been swept by the current into SRE roles.

My company SREs are not like google's, where they are rock stars swanning about doing one line fixes that make everything .48% more reliable. We are computer nurses making sure that the wards of systems we look after aren't wiped out because a doctor forgot to wash their hands after taking a poo.

However, they both require the same skills: murder mystery and the ability to dive super deep into the stack, go from python->c->assembly. or service->VFS->cache->network->time weirdness

Also it could just be copy pasting service defs and making dashboards, then pinging devs saying "shits on fire yo"

Shhhh, the union will get you.

Although sadly in most cases the automation was harder than the original service to look after.

I know a few old school sysadmins who got left behind because they couldn't upskill to DevOps and consequently lost out on $$, others who were fired after moving everything off prem to AWS because the company no longer needed them. You understand that your theory is a "conspiracy theory", so why even say it? how does that add to the discussion?

> but the tools that have been brought into the mainstream like Kubernetes

unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime.

At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers.

> I remember the days of Nagios, Zabbix, SolarWinds etc

So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser.

> The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them.

That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script.

I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013.

salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role).

They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed.

Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.

Indeed. Which is why I waited for three years to take on Docker, and two more for Kubernetes after I got burned with DC/OS, and why I entirely left the NodeJS/frontend ecosystem. I'm too old to act as the ripener for bananaware any more, and I'm pretty sick of companies exploiting other companies or, even worse, individual developers to act as their free QA department.

As someone that knows how several UNIX variants have worked since I used Xenix in 1993, those devs usually learn how a specific Linux distribution works, place them in another work, and they will be hunting for configuration files, init systems, daemons and what not.

Same applies to me, "knowing" GNU/Linux since Slackware 2.0.

I guess it's: You change the way you think about those metrics if you're the one being called at 2AM Saturday night.

Your application architecture dictates how it should be deployed and the other way around, if you try to decouple the two you're bound to run into issues. Depending on what your application does, high throughput with some loss might be ideal, or you might need at least once delivery instead while sacrificing throughput, or any other dynamic. If your deployment is done by an entirely different team from the team that builds the product itself you have little hope of predictable positive outcomes without a lot of pain.

SRE needs authority to hand back the pagers. That’s a key aspect.

To be more blunt about it: if the person writing the code is not the one on the pager at 3am, they have no serious incentive to make it correct or resilient.

This is why "you write it, you run it" is so important - the most reliable software is the software never written, but if something must be written, it's important that all incentives are aligned for it to work right.

Of course, this is only at the operational end - lack of incentive alignment is rampant in "product" organisations too.

that's great for you. at my company devops are still doing monthly releases and unable to deploy single microservices because all the services are too interconnected.

Because that is the software developer/systems engineer division that already existed 20 years ago.

The idea behind devops was to be a single team.

Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?

The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken locally and it comes down to them pinning to something like latest tag and the image “shocker” changed. But now you are fixing it and after you go remind everyone that you should pin your image to as strict as possible image tag. But no one cares because they can move fast and break things and when it does blame it on CI and now it’s not their problem.

Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off.

As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”.

Anyway thats my rant.

That’s fine, it just has nothing to do with DevOps then. The whole point is that these are not separate teams.

So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps.

People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built.

To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.)

The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it.

Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time.

It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain.

DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.

It's like saying you have a self-service waiter.

I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.

> The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.

Yeahnah, you're part of the problem.

If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed.

I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.

Agreed, many of those huge and overly complex projects also fail and overrun implementation schedules though, and they only ultimately operate with a ton of workarounds and shortcuts. I find a lot of projects that barely match original documentation limping along, or just even after a major failure or breach in my line of work, with the customer desperate to find a patch to just keep them running after a bunch of critical data is under siege within apps.

A key for me is convincing a customer to simplify goals and implement an MVP on a reliable framework, to plan versions, and to thoroughly test... Makes me a code cop, but there are reasons why these steps are in place for mission critical services, and if a customer wants to skip them and just deploy in careless "Twitter Dev Mode" I have them physically sign paperwork that states their accountability for the bad decision... When individual accountability is mentioned, they usually opt for the more stable & reliable methods of dev. Hah.

But thats just a tip of an iceberg.

What happens when your Cillium Service Mesh starts misbehaving due to missed network packets ?

Or when you have to setup a resillient backup of a cluster that runs 30 000 pods on 300 nodes ? And upgrade.

Or when you have to provide a telemetry for highly distributed system using 5 languages on landscape spread over frontend and backend with three different db engines and two clouds ?

Kubernetes is easier to start but 10 times as hard to do at scale you didnt even have back then in OnPremise era.

Yes you are right there are a lot of ppl who can write and deploy „Hello World” apps, but thats about it.

Hint, in my definition it’s none of the above

I think that once a company is big enough you can have a few options to get code to prod and make sure it stays running as expected. You could have an option where a team is focused on building most of the platform up to an agreed point (e.g. everything under an application’s container image) and provide some common ways of setting up application monitoring / logging etc…. Making sure that while there is a core internal platform as a product team that you’re focused on being a capability rather than a business unit - by that I mean you should have some if not most folks embedded (or rotating through) dev teams. You can also have an option that lets teams self organise how they get their code running - with some light touch guard rails and potentially caveats for applications with PII etc…

> In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert.

This sounds like a tale of woe. but. The fault is the culture, not the people doing the grunt work. Like devs, if you're used to nails, then you'll want to use a hammer for everything.

> gives you expertise in application design/architecture.

a sysadmin should know much better how your application runs in real life, and more importantly, how it behaves when it goes to shit.

They _should_ know more about all the different types of services (ie kafka vs NATS etc) because they will have had to set way more of them up than a typical dev. If thats not the case, you've had bad or poorly mentored juniors running your stuff.

The reason they should know, is because they have the pager to make it work at 4am.

Look, if I had it my way, and I was at a medium sized company, I would buy in 3 mainframes, in three disparate locations, set the lock step replication to "very yes" and make sure each dev used the local DB and use the job description lang, to define scale.

Pay the $1million a year to outsource reliability of DB and state. Because frankly most people can't program for either scale, redundancy or recoverability. (yes that includes k8s) let the hardware do all of that and save a bunch on staffing costs.

I work at a HUGE scale place, and yes there is a need for special sauce there. but even then not that much, and not enough to justify the massive churn in systems. Most PE is just cargo culting busy work. Stick to ECS and PSQL and just get on with making actual buisness features.

I design motherboards for network devices; I keep the internet going.

Routes are inaccessible without access to them. The underlying machine states to check for access to a route is algorithmically similar to the other checking for user access.

It’s unfortunate so many are allowed to work in IT with cliff notes level awareness of how technology works.

The layers of indirection you work with are the knobs and buttons people like me choose. You merely obfuscate with vacuous jargon to serve your career goals.

Yes, unfortunately, the misconception that DevOps is a role or person, rather than a culture, is the root of most problems with DevOps, DevSecOps, and DevSecMlTriceratops.