Six Sins of Platform Teams | Hacker News Reader

Six Sins of Platform Teams

serce.me

58 pointskaycebasques27 comments

The “internal customer” thing has been the worst driver of waste and frustration I’ve seen in my career.

It’s like the MBAs looked at the Soviet Union’s fake markets, and thought to themselves “we want some of that, please”.

My experience with platform teams is, once they are formed, they suck up the best people. Doing k8s/technical stuff.

People that should have been working on building products.

Many technical people find platforms far more interesting than trying to convert an excel sheet to a user viable product. We mixed up platform with product again after a few years as platform not only got the best people but also started to slow down the product team by taking decisions they saw as better (objectively they were) which the product team didn't agree andor didn't have experience with.

You describe the toxicity I have experienced working in the product team vs a platform team.

If you don't find it interesting to build products with whatever tool is most adequate (excel some times) maybe you should not work at a company building products. At least you should understand that you are part of the support team not the main guy.

And yes, moving to GitHub from Jenkins is maybe objectively a good choice. But if it stalls product development for 6 months, is that objectively-objectively a smart move? It could of course be, but it requires more situational awareness than "is this tool better".

Platform teams also need to make calls where the experience might be worse for some but collectively better for the company.

This is normally where you see the product engineers start to freak out.

That's the point I was trying to highlight in the "Treating product engineers as customers" sin section – the duality goes both ways. Sometimes the best thing for the company is to make the lives of product engineers better, sometimes it's to make them worse. You do whatever is best for the company, not for the product engineers.

but platform engineers are so far down the chain, it's hard for them to associate their work value to actual customer's benefits.

This is what we are discussing. A company can also choose to organize it some other way.

Yes, I agree. It was more decided to go from Product Team -> [Product Team, Platform Team] and then everyone deep tech wanted to leave the Product Team, so it reversed again later. Guess if it's from the start, you can say indeed you should not work for a company building products. Anything thing is; platforms in themselves don't really sell. The products built with them sell, so what do you work for if you want to just build platforms (want to build game engines, not games as a comparison these days).

> Many technical people find platforms far more interesting than trying to convert an excel sheet to a user viable product.

Perhaps in part because they feel, deep down, that an Excel sheet is superior in terms of user-friendliness, capabilities and ergonomics, to whatever the "user viable product" company is trying to sell? A platform may serve different purposes, so the technical challenges, even if self-inflicted, don't seem pointless.

> Perhaps in part because they feel, deep down, that an Excel sheet is superior

In my experience these folks are the exception rather than the rule. To recognize when, and when not, to use Excel vs. buying/building/subscribing to specialized software, takes a certain kind of real-world experience as a user and/or stakeholder that many engineers surprisingly lack, even long into their career.

"Domain work is messy and demands a lot of complicated new knowledge that doesn’t seem to add to a computer scientist’s capabilities. Instead, the technical talent goes to work on elaborate frameworks, trying to solve domain problems with technology. Learning about and modeling the domain is left to others. Complexity in the heart of software has to be tackled head-on. To do otherwise is to risk irrelevance." - Eric Evans

If those people were unhappy in their previous product engineering roles wouldn't they have left he company?

It is not black and white. In my experience, what people want is heavily influenced by the company culture.

If building platforms is cool then everyone wants to be there. But if building products is the thing then people want to work with this.

Platform teams could be highly toxic especially in small startups during initial secops/audit era. You really need like hundreds of product engineers to gain velocity/value from platform teams.

In my mind, platform teams wouldn't necessarily be toxic in a small-ish companies, it's just very unlikely they'd qualify to be a platform team. In the absence of cross-cutting logic to take care of, it sounds like they'd effectively be a product team focusing on the infra things?

That’s generally how they start. Most companies don’t really start with a “platform” team. It’s usually a guy who manages the infrastructure, or is the go-to for your IaC questions.

That's true, but they don't always have to evolve to follow the platform team model. They could evolve into more of an SRE-style team and/or traditional ops team.

Let me elaborate. The idea of "we have multiple product teams and they do similar stuff, let's optimize it" visits CTO/CEO brain on quite early stages. And "platform" team could evolve according to:

1) devops/sre start to provide some guides on top of some cloud, nowdays it's k8s. Like default service templates.

2) service templates transforms into custom DSL with side configuration and k8s abstraction things.

3) Abstraction/libraries on top of secrets management.

4) Service configuration per enviroment.

At with point it's all good. But startup grows, and needs a secops team to get some internal audit. Or it could be a platform team initiative.

5) Audit shows critical issues with permissions and platform team starts to think about how to restrict access.

Mismatch with "old freedom" could be quite high for unprepared product teams. Platform becomes "inconvenient". It takes huge amount of resources to make it actually usable.

I see, thank you for clarifying. That's an interesting observation, I feel like there is a story behind it. :)

I definitely agree that as any small start-up grows, the security controls become very painful, especially for those folks who felt "the freedom" before the controls were established. That said, would the picture look better without platform teams? The security controls would need to be there anyway, and I'd personally prefer to use some self-serve, platform-ish solution built by a team of software engineers that would do call auditing, verification, etc., rather than raising a JIRA ticket with some ops folks who'd do the security-sensitive thing for you.

> I feel like there is a story behind it. :)

Ahah. 100%.

Honestly I am in a great conflict right now. My current role is in a platform team (first time here, previously it was only product positions). And I'm responsible for implementing unified company wide authorization including client libraries and integration with all web frameworks in use. I feel vast empathy for product engineers having to migrate and integrate the new system. It has artificial intended bumps I have to enforce and doesn't like personally.

that's when they dont see internal teams are customers. because if they do, they will do everything to make the internal teams life easier.

enabling, not gatekeeping.

I worked in Nokia Research and Nokia Maps around the time that Apple started kicking their ass. Platforms became a dirty word in that company. Equating that to devops is a mistake. Any kind of platform is a problem.

What's a platform: anything that only serves internal customers that is treated like a product but just isn't.

Nokia had lots of platforms. Several Linux based operating systems (meego/maemo, meltemi, and briefly Android even), several flavors of Symbian (S60, S80, S90), and several flavors of non symbian legacy proprietary platforms (S30 and S40). And that's just operating systems. And that was before MS got involved and added two flavors of Windows Phone with completely different kernels. Nokia also had UI platforms (multiple). And it dabbled with online/cloud platforms as well (OVI, if you remember that).

And indeed devops and operations platforms. I witnessed what the introduction of AWS did to those. It was hilarious. First Nokia IT was outraged that people dared to use something from Amazon, then shocked it was actually cheaper and better, and then the layoffs started and that team was one of the first things that got impacted for the obvious reason that the AWS deployment was a raging success and they tried blocking that for several years at great cost to the business. Once they were off the critical path to deploying stuff, that whole unit became redundant. They had forgotten about end users and they were pissing off their internal customers to the point that those started engineering around them.

That's the problem with internal platforms: they compete with external ones and if they aren't good enough, they become a problem. You get people digging in, add internal politics to the mix, and now you have epic amounts of money being wasted.

I think Google and Facebook are good examples of becoming serial offenders on being too absorbed by their own internal platform teams. I recognize a lot of Nokia in modern day Google. Lots of headless chickens in their leadership, endless waffle about internal platforms that then flop (fuchsia being a prime example), endless reinvention of wheels (UI frameworks, compilers, operating systems, chat protocols, social networking, IOT, LLM/AI, etc.) and a long list of stuff that they discontinue because it just didn't work, and way too many variants of just about anything they do.

That's a very interesting story, can't find those in places other than HN, thanks for sharing. Out of interest, what was the reason they didn't start rebuilding their stack on top of AWS? If it was cheaper and better, then it would make sense for them to effectively migrate all teams to their AWS-plus solution? Was there some anti-cloud sentiment?

By that point they were redundant and no longer needed. And they were defensive about their internal stuff. Which of course included lots of hardware, software, network connections spanning the globe, teams that maintained various bits of software, and supported by a lot of bureaucracy, etc. All of that was replaced with AWS. That process went way to quickly for them to adapt. By the time they started complaining to management about being bypassed, the whole thing was basically a done deal.

This kind of thing is highly disruptive to internal teams. Basically it starts with them being all the undisputed kings of their territory and then somebody going first principles on all of that and bypassing it completely.

It's of course a complex story that involves multiple acquisitions and the acquired companies basically ignoring the internal stuff so they could actually get stuff done against all the silly internal bureaucracy, lack of flexibility, and internally focused & detached from reality management (many of whom were clear examples of the Peter principle).

Once one team launched on AWS, things spread relatively. The managers associated with the project got some nice bonuses/promotions and incentives to do more.

Coming back to the Google analogy here, I imagine a lot of this is playing out in the Flutter and Fuchsia teams currently. Google has flutter and non flutter UI products. And it pretty much stopped pretending Fuchsia has a future in any of it's products or internal server platforms. Correct me if I'm wrong. Just my impression.

MS is a good example that slowly got over its internal platforms after Ballmer left. If you look at a lot of their recent stuff, it's a lot less about their internal platforms and legacy and a lot more about Linux, OSS, browser standards (they killed IE even), non .Net languages like python and typescript (which they created even), etc. That's a long journey from pretending the web was going to run on IE with things like vb script, xaml, and all the other failed stuff they pushed twenty years ago.

I remember my first day working at Nokia in Ruoholahti, I was on the build engineering team and my boss didn't arrive until 12. (I arrived at 8am).

I was outraged, but informed that he had done the "Herculean task" of getting Nokia IT/Security to open two firewall ports, and it had taken 10 working days of negotiations (including the weekend, and up to 3am the (Sunday) night before my first day).

I was a little taken aback by how difficult everyone made it seem, as he really was well regarded for having managed to do it in such a short amount of time.

Before I moved to Berlin, I worked out of the Ruoholahti office as well (2005-2009) while in Nokia research. The same building had a lot of the maemo people as well. Fun place to be at the time.

I walked past there a few months ago when I visited Helsinki. Huawei has an office there now. And their presence there is not some coincidence. They were hiring a lot of Nokia people during and after the Nokia implosion. Probably lots of people in that building are former colleagues that also worked there while I was there.

After I moved to Berlin, firewall ports were indeed a big part of the ongoing frustrations with getting shit done via Nokia IT. Of course, those issues got escalated via several layers of management and then resolved anyway. Part of the friction here was that the maps unit was basically something that came into being via several acquisitions. So, there were a lot of new people trying to get shit done being blocked by silly issues like this that lost patience with all that pretty quickly.

Most of these Herculean efforts involved lots of non technical people from both sides having to have meetings with each other about a firewall rule until finally somebody high enough up the ladder inevitably told them to just "open the ffing port already and stop wasting my time" and forced the issue that way. I'm not paraphrasing here, I'm pretty sure that was a literal quote from a meeting; I know some people that got dragged into those meetings regularly.

Anyway, enough egos were bruised and solutions were found that didn't involve having Nokia IT on the critical part to anything important. Which is how AWS happened. That went from "WTF? Can they even do that?!" to "well, that actually happened" to "Oh wait this is so much less painful! Cheaper too." in the span of a few months. There was no way that Nokia IT was going to be allowed to get back on the critical path to anything.

Reply on news.ycombinator.com