Hacker News Reader

Comment by chayesfss | Hacker News Reader

Anddd, the app is still up, crazy right… like why is this news¿

This is news because later this year they are probably going to have problems. You have obviously never worked at a large internet service, but when you spend some time backstage at such a venture you learn that there is a great deal of inertia within various projects and pieces of infrastructure. If 80-90% of the team for component X departs the system is unlikely to fail quickly (a frequently repeated mistaken belief when Twitter was shedding engineers so fast) but it is going to start having problems that will not be visible to the outside world. This is not a problem for a while because the former employees who built and maintained the system were fairly smart people and built something that was relatively reliable; they wanted to be able to sleep through the night without a page or go on vacation too.

Bugs exposed by other teams using that component will go unfixed and will need to be patched around by the caller. New features that might have used component X will be sidelined because no one wants to put more load on it or risk their feature on a component that people have started whispering about in the corridors. What is worse is that the people frantically spending their days patching and repairing this Titanic below decks do not see the iceberg and do not have enough manpower to turn the ship even if they did, the people who could help give warning or helped prevent the problem left a while ago. Eventually this component is going to get stressed, some lingering bug will get triggered, or some other system elsewhere in the architecture diagram will fail and shed load to our component X that is barely keeping up and it too will fail. The danger for Twitter is that the failure could cascade through multiple poorly-maintained systems and lead to a much bigger problem.

You should visit big telcos. Things chug along not because there is a great competent army fixing anything. Institutional knowledge is regularly lost thanks to changes in tech, ever changing fortunes of the biz, divisions getting sold off etc etc. They chug along because people make do with what ever works. It sort of like your OS crashing. You keep backups, reload, reboot. You dont debug and fix. The older systems get that's pretty much the norm.

The app staying up is a testament to the robust engineering that the team did in previous years. If it'd fallen over immediately that would be a sign that the engineers did a bad job.

The fact Twitter hasn't died shows that Elon fired a good engineering team.

the other 7000 employees didnt take their code with them when they left, twitter kept that

Yeah, thinking about this as an extended vacation seems right to me. Things don't usually go down over the weekend or winter break. That doesn't mean "hey that means we only need 0 employees".

Certainly you can make a microblog service with less than 500 engineers, but there is a lot of work invested in their datacenters (from physically managing the machines to their network / orchestration control plane). That can even be outsourced as many companies do, but it ain't free.

the "story" that the employees were all terrible and the code a disaster would have been proven if the site crashed day 1

I am actually a fan of cycles where "no new features" is a priority, making systems less fragile and tollerate the weekend / holiday helps you sleep at night, and sleeping at night is nice.

550 engineers to maintain and develop the code base is still a rather large number. Is it too few? I don't know. There is little doubt that Twitter was bloated, the question is by how much.

Ok I want you to write a system that can write hundreds of thousands of messages per second, and read millions of messages per second, globally and with low latency, with every message being permanently stored and available for query. Till need you design and implement redundancy and fault tolerance. You’ll need a strategy for backing up everything. You’ll need to be able to deploy code updates to your global infrastructure which is self managed because running this in a cloud would be ridiculously expensive. You’ll need video ingestion systems, monitoring and alerting. You’ll need advertising systems, billing systems, content moderation systems, anti spam and anti bot systems. You’ll need to implement an API for developers. You’ll need applications for mobile and web. You’ll need developers to do this, testers to test it, people to organize all this stuff.

Absolutely, you'll need developers and testers.

To think you necessarily need thousands of them, though, is a stretch.

I think people have drunk the kool aid over the past decade of 'growth-driven' startup where the game was to hire as many engineers as possible as fast as possible. This is what is partially fueling the current job reduction cycle in the industry. There are simply more engineers on the payroll than needed to deliver a profitable product.

It's not going to disappear overnight, but you can see the cracks. The 2fa went down in multiple countries for hours, Oz and NZ lost access for almost a day, people reported "looping" timelines of the same 7 tweets, notification issues, and some other things I already forgot. Sure, they're up, but things are not 100% ok. The number of broken things is not even close to what we've experienced in the last few years - this was an extremely reliable service before.

Reply on news.ycombinator.com