Hacker News Reader

Comment by edgyquant | Hacker News Reader

Twitter is hosted on aws they don’t maintain their own hardware. This is a cope because Elon is clearly right. An application where prod begins to get buggy just because it’s left alone is poorly engineered. There should be CI/CD and prod should be extremely stable with the exception of non-trivial bugs that should not just start popping up by the barrel because a couple of weeks without new pushes.

View on HN

> This is a cope because Elon is clearly right. An application where prod begins to get buggy just because it’s left alone is poorly engineered.

Twitter is a top-5 near realtime web property. It has an enormous attack surface across nearly the entire spectrum of cybersecurity threats. It's hosted at least in part on prem. Swathes of it are programmed in Scala, which is a minority language that is hard to hire for. It makes money through ads, which involves significant operational complexity at scale. Its recommendations and trending features are built on machine learning models which themselves require data engineering pipelines to deliver continuous updates. It's been in continuous development for 16 years, with an accumulation of accidental complexity (aka tech debt) in addition to the ever-expanding essential complexity inherent in its functionality.

Those of us who have taken an oncall shift in the world's most complex distributed systems environments are just shaking our heads at some of the drivel in these comments.

I work in an unrelated company. Even without new software pushes and even for stable code, we need maintenance all the time.

A few examples:

- security holes discovered in the OS requires an emergency update, which requires lots of testing because updating the OS has a bad tendency of breaking even stable software (I've encountered that issue a few weeks ago);

- security holes discovered in third-party libraries/frameworks/..., which requires updating, re-testing, re-releasing stuff – and fixing whatever breaks because of undocumented changes in said library/framework/...;

- security holes discovered in your own code, of course, which also requires updating, reviewing, re-testing, re-releasing;

- monitoring your stack for misbehaviour, which could indicate a software problem (bug? license expired? SSL key expired?), a hardware problem, a resource problem (disk full?), an attack, or in the case of Twitter, one of your clients (the companies that build ads) misusing your tools and attacking you by accident;

- ... and once the misbehavior is detected, actually investigating and fixing the issue.

Sure, you can hop along for a while without anybody to handle these cases. But how long? Keeping in mind that Twitter is a high-value target for state-sponsored attackers (among others) all over the world, so any weakness will be probed and exploited mercilessly.

I run an engineering department so I’m not speaking from ignorance here. 0 days and bugs making it into prod is a red herring that isn’t part of the discussion. We’re talking about a production platform outright failing because it’s a sinking ship being kept with duct tape.

If Twitter is failing that easily, these engineers deserve to be laid off

Well, I can only speak from experience. I've encountered all of the points above just during the last 3 weeks or so. I've seen critical infrastructures in former companies being taken down by an expired SSL key or a full hard drive or a power outage or a DDoS.

Does your engineering department have a solution to all these problems that does not require human beings? Or are we talking of different things?

I would be interested to hear which web apps with 100,000,000+ users would run fine for weeks/months without hiccups if 80% of the people running it left overnight.

Twitter is not "hosted on AWS". They have started to use various cloud providers (AWS and GCP, at least at various times), in combination with their own hardware.

Even running on AWS is not completely isolating you from hardware related changes. For example AWS is retiring both old instance classes for databases and Aurora 1 soon. This is a multi week migration project with a hard deadline for me and it's not even a huge company/service. For Twitter I suspect that kind of deprecation would take month(s) to fully roll out.

That's surprising to me, given how much effort AWS put into emulating old instances types using newer instances to still be able to offer instances with the characteristics of old instance types to customers. [1]

Do you have additional information about that you can share (maybe an announcement regarding this made by AWS)?

[1]: https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-n...

It happens quite often. I'm dealing with deprecation of db.r3 for example which will be enforced in a few months. (I easily find the public announcement - you get an alert if using it) The instance sizes don't cause that much problem as such, but you still need to spend time assessing the new performance, updating reservations, figuring out the transferability of the old ones, etc.

Reply on news.ycombinator.com