978 karma · joined November 11, 2018
https://kenschwaber.wordpress.com/2013/08/06/unsafe-at-any-s...
> Keep the values, keep the principles, think for yourself. A core premise of agile is that the people doing the work are the people who can best figure out how to do it.
Teams have agreed to use postgres for sql, this is by agreed convention.
A tooling team implement hashicorp vault and build a pg module/deployment that is packaged and accessible through whatever tooling you build.
This makes a datasource of type pg sql available to all teams in a couple of minutes.
These modules/deployments are governed by a bunch of conventions, but those are of no special concern to consuming dev teams.
Use “something” to keep track of service upstream/downstreams: consul, a database, or something else.
When a service is connected to a database, say using consul upstreams you have the job gen api inject the appropriate env vars according to an agreed upon convention.
Now dev teams have an option to deploy incredibly secure pg databases, and by simply setting a service as downstream it will automatically receive rotating connection strings.
There’s no configuration to be managed by dev teams except keeping a service graph up-to-date.
Lot of conventions above, and a tooling dev team with a lot of infra and tooling expertise.
I believe that for scaling dev teams you should look at tooling like a product domain and treat it as such.
Larger team(s) using free form yaml configuration and templating quickly becomes messy.
Setting a boundary at “configuration” and exposing configuration options to dev teams through settings works really well.
It requires you to have a tooling team with a few developers, but it scales!
When discovering gpxe/ipxe it got real interesting - now we could make web calls in pxe boot mode that let us automatically select os image based on asset tag from our cmdb. Role based app deployments to servers, and app streamed to clients. Those were the days… We built ci/cd not even knowing it was a thing.
When cloud started to be pushed I was like: “so pretty much what we’ve built here” (barring the promise of elastic scaling - we had barcodes preped from vendor batched to cmdb before delivery though). I wasn’t yet aware that most places didn’t use software to manage compute, end to end.
We’ve started to use it (temporal) a bit for general automations, and it’s pretty great. Monorepo with a lot of different activities (“microservices”) makes sense.
The activites are orchestrated in workflows (much like DDD “sagas”) and scheduled via temporal. This gives awesome introspection and observability.
I was on a team that ran performance critical stuff for a reasonable sized ecommerce business - moving from vmware to bare metal was fantastic on every level from api’s to performance and reliability.
The enterprise spirit is stong in vmware.
The teams and management structure will immediately become a technical debt.
If we let the product ”decide” where boundaries actually exist and team up accordingly there’s a chance to scale and maintain a bit of velocity.
It requires constant introspection, monitoring and scrutiny though. Something I’m constantly thinking about is how to scale that beyond 20-25 developers. Gitlab have a nice section[0] in their handbook on releases and flow of small bits and pieces, and internalizing something like that together with clear domain boundaries could be a ticket.
Basically - never try to resource optimize, always figure out what good flow looks like and find ways to keep it flowing.
[0]https://about.gitlab.com/company/culture/#freedom-to-iterate
I have taken my kia e-niro on 1000km roundtrips, and this is totally acceptable as you have to stop for at least one meal, possibly two.
The convenience of an EV is fantastic - but I’m lucky to live in a building where charging is available.
This is an example of an actual goal I have for members of my team - it is Specific, Measurable, Attainable, Realistic/Relevant and Time-bound. It's also tangential to the OP's topic here in a couple of ways.
Good things happen when you allow for slack, but we often put too much pressure on ourselves, and won't allow it.
I'm looking at it as a bit of "lucky lotto":
https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chao...
Rule of thumb: are we, or someone else, going to second guess this decision in a year or two? If it feels likely an ADR is in place.
I must say that using nix on my primary workstation makes it feel like an appliance. The system generation roll-back have saved me a lot of times when an application update brings a regression.
Prolonging life and reducing negative health impacts are another bonus.
Yes, there is still an environmental cost of the production of EVs but these vehicles don’t emit CO2 or greenhouse gases and so their lifetime carbon footprint is much lower.
For example, a recent Reuters study found that a Tesla Model 3 would need to be driven for 13,500 miles (21,725 km) before it does less harm to the environment than a Toyota Corolla:
https://www.reuters.com/business/autos-transportation/lifeti...
I am, almost an involuntary, manager of 6 different dev teams, and I often say that my aim to reduce the “management” part of my job to a shared duties whiteboard.
Having this as an utopian goal I really strive to remove anything that will hinder team autonomy.
Transparency, communication and ownership is crucial to this mission. Always team, never individuals!
The diary entries are supposed to replace other forms of communication - not be an addition.
My place of work strive for autonomous teams within specific business domains. You describe yourself as kind of a bottleneck, as things are expected from you in various projects, via email no less.
We’re facing completely different work environments, and as such you see a daily or weekly log as an additional burden. I see it as replacing tones of emails and meetings.
Again - it’s not required here, but as people have noticed the use and asynchronous discussions arising from these log entries, it’s taken a hold. I do them myself as a manager, as my team deserves to know what I’m up to.
This is probably true if you work in complete isolation.
If, however, you are a part of a larger team or organization a lot of value can come out of this type of transparency - it removes the need for a lot of meetings and is a great tool in an asynchronous work environment.
Just two minutes ago one of the devs in a team I manage picked something someone from a different team logged about, and decided to initiate a discussion about a potential conflict.
I agree that it should not be mandated though, but should come from the fact that people see it as valuable.
There's also a cultural aspect of it - it brings life to an online community that have to be built, and it works to fill the need of casual interactions during lunch or other spontaneous gatherings that no longer occur with regular cadence.
It’s often a licensing issue, and definitely a security issue.
Either you maintain this manually - proper time consuming chore at larger companies with many systems and applications.
Or write automations to manage it. Better, but still a lot of work and not always technically possible.
Or you hook as many of these systems as possible up to an SSO solution backed by some kind of identity provider.
This grants many benefits for everyone during the lifecycle of systems and its users - sysadmins, it-support, infosec but perhaps most of all the end-users.
As a manager myself I first and foremost think about how the on-boarding works: is it smooth, and are new hires gaining access to the systems they need without 15 calls to the service desk.
You generally want domain expertise shared in a team, and continuously molded to perfection over time.
Mobs and pairs can really shine here, but it all depends on the individuals.
In teams where it works well it’s an enormous value add, in my experience, and the hive mind you reach after a year or two within a team is fantastic.
As a dev manager I let the teams decide, but looking at team flow over time, it looks hard to beat a well functioning pair or mob team. IMHO & YMMV, and so on.
Having managed many thousands of hosts over the years, I’ve longed for something like nixos. Managing largeish scale infra is not what I do these days, but nixos still appeals to me, and I use it for a few pi’s, a nas and my workstation.
I’ve never tinkered less with my machines, and I love it.
With nixos ad-hoc tinkering is not “the way”, and using a shell.nix for different project folders is just awesome. I imagine this could help scale dev teams as well.
It’s about scale mainly.
A nixos host can be viewed as a solid state environment, in some respects. At least from a systems management perspective.
If you scale up to more than one host, that you want to keep in a completely manged state, nothing really beats a model like nixos (or guix, although I lack experience).
Having the complete system state of n machines, where n > 1 nixos is hard to beat. I have extensive experience with tools such as ansible and salt, but the state management of nix is just unbeatable. It’s, IMO, “how it’s supposed to work”.
And with nixos you usually don’t have to nuke anything - a new system generation is enough.
It all comes down to measuring outcomes over time. Doing agile is doing right-ish things at the right-ish times, including ways of working.
Use retros - are we being efficient over time? Why and when? Why not and when not?
My teams tend to flow in and out of pairs, mobs and single contributions, all depending on task at hand and circumstance.
Flow! Are you flowing? How do you know?
Over time I’ve noticed that really well gelled pair and/or mobs deliver more consistently, over longer periods of time. YMMV.
He’s incredibly far from a normal human - he’s a true specialist.
Lots more energy goes in to that, compared to losing excess fat and living a “normal” human hunter gatherer existence.
The whole system change once you go google scale, so to speak, and phelps is gmail.
In the words of Ido Portal: we’re human first, movers second and specialists last.
I see relevance in this research in the human/mover perspective.
Ido Portal, for all his eccentric weirdness, knows things:
> The body will become better at whatever you do, or don’t do. If you don’t move, your body will make you better at not moving. If you move, your body will allow more movement.
I’m personally thinking in terms of micro-habits that eventually form a lifestyle, and it’s all about total lifetime reps, if you will.
For example, I have a height adjustable desk, and have replaced the chair with a pilates ball. I still stand, or sit on my knees, more than sitting down on it.
I believe there’s a problem with some of the ergonomic setups we use - if you keep moving, and shift around you shouldn’t need it. If you do, you’re already “losing it” and should probably be changing things up more frequently during the days. Now this is easy to say, and for some people, probably not true.
I’m basically continuously fighting the path of least resistance. I’m in war with the comfy a* brain!
Judo, with it’s focus on grappling standing up is ridiculously hard on the body, and is not something you just pick up late in life and chill with. You’d break in all sorts of ways.
Carmack seems to have a serious interest, starting with wrestling in highschool.
> My wife for Christmas once got me a year of privates with Carlos Machado which put me up several levels. There was a period there where I’d have Judo with one of my coaches one day, then privates with Carlos the other day.”
https://bjjtribes.com/john-carmack-details-his-bjj-and-judo-...
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Very good post, that holds a lot of experience. Unfortunately these things have to blow up in your own face, often more than once, for a lesson learned.
> It’s a bit smarmy of me to criticize them for waiting so long between releases. They didn’t do it on purpose, now, did they?
Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make:
They decided to rewrite the code from scratch.
This is why startups or VC backed things can explode, and innovate, but once you IPO, innovation is not what’s important - it’s about perspective I guess.
When the death spiral of anti-innovation starts, people such as talented developers start to drop off, and in the end you’ll have a slow, bureaucratic mess of an org, sustained by the inertia of an already existing customer base.
“How about we yank up them licensing fees?!”, or my favorite “Let’s rename a product suite!” (Citrix, as a case in point, is notorious for this…)
This be the enterprise IT lifecycle.
- ownership and its boundaries becomes blurred
- maintainability over time
- lack of systems thinking and architecture
These are things we struggle with in organizations with dedicated development teams!
When parts of critical apps or workflows become like “xls in sharepoint” - PoC’ing is dreamy, anything beyond that potentially very painful.
Just my 2c, although I’m keeping a watchful eye on this segment.
It doesn’t make the infra technology more or less interesting?