Hacker News Reader

It's incredibly difficult to fund internal platform teams appropriately. Usually one of three failure patterns emerges

1) The team is competent but picks up migration work to arbitrary technologies, and approaches with no clear ROI. These migrations block feature development and never seem to end e.g. teams ceaselessly migrating from GCP to AWS, to kubernetes, to podman from mysql to postgresql etc.

2) The team is operationally heavy and generates arbitrary requirements for everyone else to follow. The toolchain seems to get worse over time, and the number of hoops to jump through to get anything done endlessly grows e.g. Wait 2 weeks and get three business approvals for a server which you aren't allowed to have root access to.

3) The team has big ideas, but the business constantly under invests. The team is called to fight every fire but unable to stop the fires through any meaningful project. The company ends up on a platform thats constantly on fire.

When weighing these execution risks building an internal platform for just about anything looks incredibly expensive. I've only been at 1 company out of 6 which nailed the internal platform tooling requirements. The only reason I can attribute to their success was through quarterly NPS surveys on the developer experience for every major piece of the companies toolchain + hard to meet SLAs for uptime.

I think you need to also include the reality that you graduate to 2 to stop the bleeding from 1 and 3. I’ve been at companies where we have had literally no choice but to implement 2 because the ops team was drowning due to every part of the business demanding to be priority 1 and not being able to actually reason about prod because devs would manually make changes.

The underlying symptom is bad project management and more work than the team can handle but convincing the business that the ops team actually needs to be twice the size and have dedicated managers falls on deaf ears.

As you say, this de-skilling problem is broader than SaaS, it is tied to oursourcing of critical competencies or, in younger companies, never having them in the first place.

If you want to see de-skilling in action, hard core, go look into the service providers, wireless and fixed. They are running on fumes and attrition-victorious teams that last had a new technology in the late 70s/early 80s because everyone good at networking went to the FANG predecessors and then FANG proper.

It’s also a problem with compensation. Many companies take an incredibly stupid and paternalistic attitude that employees should be happy with 3% - 5% raises, no bonuses, and refresher equity way less than their original grant.

Anyone actually talented just leaves after 2-4 years, so you get zero cohesion or snowballing effect of talented tech leaders who other good people actually want to work with.

These places become total career death. Eventually it becomes a place where only people with kids or other significant life obligation go, specifically because they value no-ambition bureaucracy that trades off compensation and risk taking in exchange for work life balance.

Those places are so soul crushing. Avoid at all costs.

> we're going to lose the people who know how to design and manage petabyte scale hadoop clusters.

Why is that different from "lose the people who know how to design and write accounting systems from scratch?" That was what happened when packaged accounting systems showed up. I'm not sure why you would want to preserve knowledge of Hadoop if other technologies are more efficient.

This is a good point. By the same token, most of us have no idea how to milk our own cows or forge our own knives. Having businesses dedicated to one craft and providing their services to those who'd rather be doing other things seems like a natural model humans - and nature itself - have been following for millenia.

It's not like stem cells worry about losing the ability to excrete bile when they become neurons.

This happens with every technology stack from steam engines to software. There was a time when programmers could solder together an ALU using transistor gates.

Well yeah I could probably have done that with some 7400 series TTL ... but the MIPS wouldn't nearly cut it in today's world :)

It might be the same with 'trains' (steam engines); I don't know much about that. But losing skills in 'the cloud' makes you prisoner to PaaS/SaaS providers; if these skills almost completely disappear (let's say, akin to your transistor gates, that almost no-one can install or maintain server setups anymore), the cloud providers really can do whatever they want price wise. For instance, if a cloud provider misconfigures something at scale in this great future, there is no way for anyone to verify that; it might be that you are paying 1000x what you should be paying for a particular database query that triggers a perfect storm in your set-up. As no-one in your company knows and no-one can check what or even if this is going on, it'll be considered 'normal' and you just pay. This is already happening, at scale, but not due to the cloud provider's fault, but because people are already not learning skills. If I would have $100 for every missing index I find per month in client's setups, I would be working a few hours/month for a very nice wage. Heck, most startup databases I see these days don't have any indexes at all; and then they wonder why they are paying $10k/month at aws while I can get them the same or better performance for a few bucks/month on a vps. Not because I'm so smart, but just doing everything 'low-level' wrong because no-one simply knows anymore how a database works (it's the cloud! we shouldn't have to!). It's not restricted to relational dbs either; sometimes I cry when I see dynamodb setups made by people who think this is just the cloud way of working; 'throw in whatever data, it'll always optimise itself'. So this is already happening at many companies and it's getting worse.

Similar thing happened in the past when application developers (most apps were written assembly in the past) built a dependency on instruction sets. If the computer manufacturer underperforms in future years, you are stuck. This dynamic can never be erased even as we keep climbing higher on the stack.

Another thing I noticed that adds to this point; the investors (well their tech DD people) seem to really push this. I have seen more and more teams not advancing to the next round because they wanted (with good reason imho) to engineer their own tech than 'just go with aws' blindly. AWS costing literally over 10x (that's lowballing, more often 20-30x and sometimes 100x) as much. As a startup, that's not always the best choice imho, but I see people getting forced into it as investors (again, their tech dd peeps) reason that it scales better than people (it does, but that's usually not needed at all at seed and even far beyond for many types of businesses) but in that, they lose sight of that these skilled people might not join at all because the money you could've paid them in the early days to get/keep them on board, now disappears into the cloud. It seems most investor's (here in the EU at least) dream to run companies with only MBA's and this is a solid step in that direction.

Using third party tools doesn’t lock you out of them. Nothing stops you from the collecting that data before sending it off.

Every company outsources something fundamental. Does Google mine its own metal? Generate its own electricity?

Even if you did it on prem, that doesn’t save you from license renewal costs or upgrades. You can write the software yourself, but that’s not cheap either.

Is metal and electricity Google's core competency though?

I don't think anyone is arguing one should maintain their own silica, atoms, independent universe, etc .

Using Hadoop doesn’t mean your core competency is maintaining Hadoop. Analysts don’t want to be engineers.

> independent universe,

Solipsystem, Inc. begs to differ. Having it your way is a booming business nowadays.

(j/k, borrowed an SF title https://en.wikipedia.org/wiki/Solip:System )