978 karma · joined November 11, 2018
I use to say that it’s like one of the business domain development teams, it’s just that the domain is “dev”. The businesses devs are the stakeholders, and the tooling/platform team members are usually already domain experts that write code.
Having a “devops” team would feel like having a team called “agile team” or something - you do “devops”, you can’t build it.
This is also a great way of scaling teams - keeping a path of least resistance that is secure and compliant while giving observability and conventions.
Plowing through this atm, about 10 months in…
In my experience, layers love to get intertwined in, so called, standard systems.
You end up with a business not really in control of its processes and developers who are experts in non-business domain data models and dsl/code abstractions.
What you wrote really hit it on the head for me in my current position.
Someone wrote, perhaps Tom DiMarco: “a process is per definition not agile”.
I seem to go through these cycles with 3-4 weeks of great sleep and roughly 2 weeks of getting stuck in N1, observing.
Really interesting things show up while in this state, but it’s thoroughly incompatible with three young children…
Too many processes are never automated, and people are assigned to manually enforce and follow up on processes, that should be completely automatic.
This is where process busywork creates yet further busywork. Prime example? ITIL (as this is an IT focused site many probably have heard of it).
If you don’t automate the heck out of ITIL you reduces quality, slow any pace down and the mess remains.
Lessons learned over and over: Working code works, and, don’t let perfect be the enemy of good.
If you can change it with little effort it’s as close to perfect as it have to be.
Ofc not all domains are the same, but here I am with quicksand in my hands.
What I mean is that a fs snapshot is ”dumb” in and of it self.
If you couple zfs with the nixos rebuild command, then… sure, I guess - but the previous generations are already more or less directly available, unless GC’d.
This is what the fine manual has to say:
Since Nix is good at being Nix, most users will want their server's data backed up, and don't mind reinstalling NixOS and then restoring data.
Then there’s the question of how exactly you reached this state. Having nixos generations is like having an event stream of all changes. Apply your backup to a new machine, what happens? Who knows.
In nixos/guix it’s not about only ”package tools”, it’s about treating the complete system and its state as a coherent whole.
Once I got the taste of it I see no way back to opaque packages, managed by config tools with no concept of state.
And if you’re a dev - shell.nix and declarative containers ftw.
Having supported large enterprises and pieces of the movie production industry I can tell you there’s a vast difference in how end-user IT is treated.
The users you are referring to are power users that get to select their own tools, more akin to developers (at decent places at least).
Currently a dev manager, about half of my dev team want to use mac. They can, with zero support from central IT.
That’s not a choice our sales org have, for example.
There’s a lot of effort that goes in to support and partner channels for enterprise offerings. Making servers seemed easy enough? Look where that ended up. It’s a completely different business.
The app instance authenticates using the cloud providers key vault and from here it’s allowed to use a policy claiming tokens granting rights to a specific db role. These tokens are short lived and are automatically renewed.
This is just one case - vault have a lot of useful integrations that let you “float above” any cloud:
https://www.vaultproject.io/docs/secrets
I like to keep my concerns separated. That way we can use any cloud as IaaS and keep the platforming part ourselves. Vault is at the heart of achieving something like this.
Service discovery is in part DNS with low ttl, but there’s a part of it that ties in to configuration management, monitoring and observability as well.
If you “update dns” that means you’re doing a part of what consul does (only f ex nomad does it for you). Nothing magic, but you still “need it”.
Tags and metadata is a large part of it though, as well as configuring things automatically based on state of the SD and K/V.
In the end - whatever works, I guess.
Also keeping discovery and configuration separate from the cloud provider makes a hybrid approach feasible - which I believe is relevant.
Big cloud pricing is big, depending on circumstances.
I’ve yet to set foot in a bigger place with distributed teams and services that only use k8s and nothing else.
In these settings there’s a need to interconnect and monitor everything.
I’ve even at one time registered mainframe stuff in consul, as an external service, with a rather complex health checks in place. This alone improved observability and reliability for a lot of the distributed consumers.
Keeping services discoverable, with service health-checks and configuration data at hand in the k/v is not needed in the “cloud”? I guess a lot comes down to how you opt to manage you services… It’s what etc does, but worse (imo), for k8s. My usual work with larger infrastructure spans more than k8s or a single provider, hence consul is a given.
To my knowledge no other secrets solution exists that cover all the things vault does, and at the same lets you stay provider agnostic. It integrates well with the major cloud providers though!
- Vault is not niche - it’s THE way to manage pki and credentials if you’re half serious about security. Which is why you’re now are starting to see managed vault.
- Consul - EVERYONE should use service discovery, cloud or not. It’s indispensable for numerous reasons. If you doubt it’s relevance, check out the Kubernetes integration work - there’s a reason for that focus. You need service discovery if you operate at any sort of scale, spanning multiple providers and teams (Azure have a managed consul offering btw).
- “Trust” nomad? The team and I have used it since 0.4 and 0.6 in full production at two different companies. K8s as well, but it lacks the unix vibe of “one thing, and do it well”, which is something you get with nomad, consul & vault. Nomad has been rock solid and I’ve so far had no reason to not “trust” it, 100s of thousands of deploys later.
- terraform spans many providers. It’s a good tool, not without it’s quirks. But I’d rather have one quirky tool than multiple quirky vendor ones. Also, we use TF for basically everything - even the stuff we host in-house through lxc and postgres for example, and through home grown providers as well.
I could write pages on the hashicorp products!
Renault Clio on spot #2 with 18k sold.
Interesting considering the price of the tesla….
If you declare the business process you want to support - either in code or in BPMN, you're automatically mapping out the events that are relevant to the business.
Having a system that takes care of executing this, and tying the pieces of this workflow together is one example where ES can shine IMHO.
[1]https://docs.camunda.io/docs/product-manuals/zeebe/technical...
The word ”traditional” have also been thrown around.
For me ES makes perfect sense in systems handing a workflow for you. Products like Zeebe[1] and Ubers Temporal[2] comes to mind.
These systems basically becomes your business transaction log and as such ES is just a great fit. (I have no idea whether ES is used in temporal though… never looked at the code)
This is a big one, IMHO! The cost of this will be for my children to deal with.
Also - a normal adult, not pursuing an athletic career need about 120g of meat protein A WEEK.
> If I’m a leader that is serving a team of people doing manufacturing work, there are scientifically proven processes and techniques that can be readily trained and applied. And someone who learns those manufacturing management techniques can reliably boost team productivity. The same cannot be said for software leaders
Leadership is one thing, process of work something else. Using techniques from manufacturing is one of the most effective ways of improving output and quality of a software team. The Lean principles matches up quite quite well with the work of producing software.
This, however, does not in any way imply that you should treat devs (or people in general) like machines.
Work process is incredibly important to create a professional and able organization - leading people here without infringing on their motivation, continuous learning and control is what management is about (to me at least).
Many of my young devs have not lived through the 90s and 00 with it’s projects and outsourcing/off-shoring strategies.
They’ve landed in the aftermath of an agile revolution turned a beast in it self. They’ve seen nothing else and are forced to accept it. My job is to have them level up and pick the cherries out of the mess we’re in. Have them make it theirs - have them own the process.
Agile, in my mind, is about professionalism, ownership and accountability on behalf of the devs. Give them the freedom to claim the above and an insane output can be achieved while having fun.
Simple but clear process should surround this work, but the ”why” have to be internalized by the team.
I’ve been places with many different tools and people.
Guess what, it’s ALWAYS about the people.
If a team is following principles derived from or resembling the manifesto, the tool will be shaped accordingly.
Jira, as an example, sure can be shaped, and I’ve done it myself several times - remove cruft, keep it simple, change if needed.
A tool is not responsible for this breakdown. Lack of clarity around ownership and responsibilities is, by the look of it.
IM humble O.
Why on earth would a manager be allowed to set tolerances like how you describe, tool or no tool?
It makes no sense and is first and foremost a management and cultural issue. Second a work process issue. Solid third, one of competency. Probably ways below numerous other problems lies the tool.
You obviously needed to be able to set tight tolerances for some work, so… you seem to need the setting on occasion.
These stories just blow wind in my sails!
If you have management like this (american?) you must be measured like crazy - set goals based on metrics that will push things in a direction you see effective.