DevOops, some common anti-patterns
hackernoon.com
hackernoon.com
I witnessed AWS creeping into Nokia ten years ago. This did not go down well with internal ops initially. They did not realize they were part of the problem instead of the solution. They were a huge cost bottleneck and also a source of huge time delays and pointless bureaucracy.
Ultimately a lot of what they did stopped being a thing. Resources they controlled (e.g. firewalls between data centers, storage clusters, vms, etc.) were replaced by IAAS. This turned out to be magnitudes faster, cheaper, and more reliable (e.g. storage cluster outages were a regular thing).
Small organizations don't require dedicated ops teams anymore. If (dev)ops is done done well, ops is just not a full time job anymore. However, letting development teams do their own ops requires hiring people with ops skills as well as developer skills: i.e. devops people. So, not an anti pattern and it is perfectly valid to make sure that all teams have enough ops skills in the team. Also, these days I would expect senior developers to be not completely helpless doing ops. I don't expect them to like it but I do expect some rudimentary skills.
So,hiring, people with devops skills is not an anti-pattern. I'd actually argue having an ops team at all is an anti-pattern unless you are big enough to justify needing one. And if you don't have one, not hiring people with relevant skills would be a big anti pattern as well.
I've been around long enough to have seen most of what ops used to be about automated away or replaced by IAAS style deployments or otherwise no longer relevant. IMHO this is an ongoing thing and there is still a lot of non trivial stuff involved. A lot of time is still wasted on what is fundamentally tedious, repetitive, and error prone business of reinventing the same wheels over and over. It still seems way too hard to push out bog standard web applications in a way that results in responsible deployments with all the boxes ticked regarding security, ease of use, logging, auditing, etc. For me devops is a necessary evil. The less time I can spend on it the better. But the reality is that it still requires skills and time.
Ie, I advocate self hosting a lot in my ~8dev team (in a real, 10 year old small company). I advocate this, because the last company I was at was the traditional startup and we blew tons of money on cloud solutions. Ie, if we needed to aggregate logs, we threw everything at papertrail, rather than managing it ourselves. This has given me a cost aversion, so I theoretically prefer self hosting when sane. Mostly for everything but actual VMs.
With that said, a lot of self hosting leads to a lot of infra. I agree that hiring devs who also know ops is a good way to go (and would fit us well), but in this scenario, do you still think it's the better/best?
edit: to be clear, I advocate using AWS/etc. I don't advocate using dozens of SaaS options to solve our modest needs at non-modest prices. Many SaaS solutions were designed for scales much larger than ours, self hosting seems most apt.
So, full-disclosure: I come from Ops. These days I'm more an orchestrator/developer than a hardware/bios/cable monkey though. For context my last deploy was 2,500~ physical machines. So I'm not the kind of ops that the parent is talking about.
Now, if you don't have the money to do cloud, and you don't have the headcount/experience to do self-hosted properly; then you're _done_. You will accrue technical debt at an exponential rate. At my last job Ops on the production environment was done exclusively by developers.
When I arrived it was an absolute shit-show nightmare. Silent data corruption, weird dependencies on services which were not highly available, hand-built machines, next to "cattle" but which had multiple deployment scenarios (capistrano, chef, bash, PHP kludge) none of which were documented.
But the worst thing was definitely the silent data corruption.
I agree in part with the parent, Ops is a dirty word. In my current company there's a separate "Ops" entity which is a true horror show exactly as he describes. However having people trained on ops in your team is -absolutely essential- to doing devops correctly. Developers and Operations think differently. It's a gross generalisation and there are exceptions but by-in-large developers are focused on "how can I make this work" (even robustly), where operations are focused on "how can this break and what are the implications if it does" and those methodologies are intrinsically opposite to each other. A defensive programmer is a great thing but they are not only rare they are not a panacea.
--
Pick a side; do cloud or hire an ops.
Cloud is basically "ops as a service", giving you all the tools you need to get by efficiently without classically trained sysadmins. I really promise that you will not get very far without someone handling this side of things for you, whether it be the SRE's at google/AWS or someone you hire.
I'm talking about not using dozens of SaaS options for small tooling, think CI, log agg, metrics, git, etc. Much of these are trivial to self host when your traffic is small and predictable. Thoughts?
Edit: Just realized I didn't answer your initial question in the post: A cloud architect, who comes from an engineering background, is more along the lines of what you would need versus just a pure ops person. My last company was a small company (startup mentality) and as chief architect I was able to design our cloud architecture as well as write code on our projects when my developers were struggling to meet deadlines.
Ie, I don't advocate self managing hardware. That's a difficult problem, one more cheaply and easily solved by using AWS, GC, etc. Yet, there are dozens of SaaS options for so many little problem you could have, and they can bleed you dry. A balance is needed.
I mentioned in my original post that I advocated self hosting except for VMs. Ie, I promote usage of AWS/etc. I just don't think we can afford Papertrail, and the billion other SaaS pieces I used in my last company. Self hosting has been far more scalable and cheap, due to our modest needs.
Log management in particular is really a shell script/self hosted app kind of task, just share the money saved on Papertrail with the team and I'm sure they'll get on board.
So who's gonna deploy the developers' applications and implement the necessary infrastructure for that application?
Who's gonna create, maintain, and administer the CI/CD pipelines which allow us to do testing, receive feedback loops, and rollback when code does not work?
Who's gonna be the guy, or team of people, who has knowledge of all the different services and architecture of all these IAAS? And then who's gonna know how to implement that?
I have yet to see a dev with good ops skills. DevOps are not dev that moved into ops world it's quite the opposite. It's Ops people that moved into dev teams.
The other thing is also that dev usually don't want to do that, and when you force a dev to do ops things it gets very dangerous.
Changing teams to get a break from being on call starts to seem less disruptive than staying, and I'm sure plenty of other people where I work hit that realization long before me. I think more of a "pair programming" approach would often produce superior results. Two close associates don't need to know every command-line flag and semicolon of each others' work to understand and appreciate each other's contributions to the challenges they share.
But as I always say: "Devops is not a job title"[0] only cheap hiring managers think it is.
[0]: https://www.infoworld.com/article/3263812/devops/do-not-hire...
I'm not trying to argue that it's a silver bullet but there's a bit of circle jerk over k8s complexity when it's not that hard.
This is basically the "hit-by-a-bus" scenario and it applies to pretty much anything in which you have a single point of failure in terms of manpower, not just in-house software. e.g. What happens when the engineer who installed your open-source solutions leaves? Who is left to support updates, migrations, etc? The solution to this is of course is to staff correctly...so that you don't have a single engineer building out an entire code base or system that your company heavily relies on.
The only difference is server resource management, which is not much effort if automated correctly, and security fixes which is not that much more.
It's like some companies had testers, but then they turned them into developers also, and at the same time everyone becomes an ops person. What could go wrong?
Irrespective of personal feelings on these individual projects, they're all aimed at automating something?
I don't get the point of this item?
What's wrong with them?
The tools are good, but use them when you need them and when they help, not just because someone told you it was best practice.
I implemented Puppet to manage a hugely complex application/environment. Because the architects of that app didn't think of Puppet, they threw up roadblocks at every opportunity. Things like refusing the allow the install package to be simply changed to RPM, or allowing a large part of install logic to be moved from shell scripts to Puppet. Or allowing an automated push from the release branch of the Puppet repository to any of the Puppet environments.
A lot of stuff that was initially planned that would have greatly simplified install, configuration, rollbacks, verification, etc etc, was all refused because it had been invented by the ops team instead of the architects.
So we ended up making things worse because management wanted Puppet, but didn't want to lay down the law to the architects to update their stuff.