Twitter engineer claims problems will pile up, leading to a breakdown | Hacker News Reader

Twitter engineer claims problems will pile up, leading to a breakdown

dailymail.co.uk

6 pointskiyanwang9 comments

I tend to think this is overblown, if for no other reason than, whatever the team's current status, there are always mercenary, high-rate consultants that can be brought in to help.

Maybe it's overblown, I don't know if consultants are really the fix though.

What kind of consultant jumps into a completely new codebase of a massive distributed system with hundreds of millions of users that has broken down to the point of being unusable and manages to fix anything, and how long does that take?

Sometimes it takes months for regular new employees to do anything productive in some of these codebases.

> Sometimes it takes months for regular new employees to do anything productive in some of these codebases.

Which is most probably why most of them were just kicked out. Twitter isn't complicated, it's literally just a micro-blogging platform.

I think you're underestimating how difficult operating at that scale is. There's no "just" here. It's hundreds of millions of people, all over the world, using your app 24/7, with the expectation that your platform is always there, that it shows consistent data and that it responds instantly. That's very difficult to build and maintain.

"It's just a banana Michael, how much could it cost?"

I get paid to jump into completely new (though mostly old) code bases that have problems. But those are mobile apps implemented on a tech stack I'm familiar with. Twitter-scale is totally different. I have a family member who works at keeping major parts of a major platform running. At that scale you make your own tools. Nobody outside the company is ready to use those tools without training.

This. If there's no one left to pass on the tribal knowledge, the contractors have to reverse engineer how that stuff works. It'll take months to rebuild that tribal knowledge.

The underlying tech is probably standard (e.g. they probably haven't invented their own routing protocol), but the control plane is likely custom. The impact of making manual changes or changing the control plane is probably unknown, ranging from being fine to borking the system in some way that even reverting the changes won't fix.

The best option would be to try to rehire the former employees, probably at dramatically increased pay. I don't even know if that would work, though. Bridges have been burned.

I have no doubt the consultants would be less efficient (perhaps by orders of magnitude) than employees who'd been there for years, but I think they could at least keep the lights on.

Already started: https://mashable.com/article/twitter-copyright-full-movies

Reply on news.ycombinator.com