Rearchitecting Core Services at X

Because you have a hammer and so everything sounds like a nail.

When you discuss a problem at a high enough level of abstraction, then yes everything looks like a database, and at a high enough level of traffic all databases need to be partitioned/replicated.

The much more interesting question would be "why doesn't the thing I'm familiar with solve this problem"?

My hot take is that 90% of software engineering is just transforming data from one shape to another and there's only so many way to do that.

It doesn't matter if you're syncing between a database and server, between a presentation layer and an object model, or across two devices over a network.

Fundamentally you're dealing with an impedance mismatch that only has so many types of solutions.

It could be that they're plowing billions of dollars into engineering and missing some obvious solution.

I think it's more likely that once you get down the details (which are going to much more intricate than a post like this can convey), then it may turn out to actually be a hard problem that requires a bunch of engineering.

For a relevant example of the "how hard could it be"-mindset: See George Hotz's short stint at Twitter.

Doing 60 million queries per second on a partitioned database isn't going to be easy either, and when you have problems you'll have to dig into the code of a database you didn't develop.

There's a lot of finer details that influence how you design system. Yeah reading data is just SELECT, but there's too many things to consider.

- Volume of reads: how many people are reading tweets? - Volume of tweets: how many unique tweets are being served to people? - Data skew: Elon's tweets see hundreds of millions of reads, vs some tweets only see like 10. - Size of data: A tweet looks like only a few characters of data, but every tweet has probably hundreds of columns of metadata.

These are just a few considerations. Plus, scaling out replicas isn't always the feasible after a point. Otherwise, why not just spin up 1 server per tweet and keep things simple.

Because they make billions, and they have to spend it on software development.

That these software devs then go out and build slightly different offshoots because of business requirements and technical requirements is, in many ways, incidental. Devs at smaller companies end up writing poor-man's equivalent of various techs all the time, or write interesting things in their spare time. It just doesn't get the same attention because they're not as big.

It's how they get paid. Replacing bloats with extremely simplistic equivalents like stock MySQL instance isn't going to convert well to paychecks. It also means your output will be erratic(because it'll be opportunistic), architecture design will be less resilient, and you will be a brilliant jerk that are hard to anticipate.

Elon Musk was supposed to be an uber-genious charisma who've overcome that problem. I suppose he no longer is.

Reminds of this: https://news.ycombinator.com/item?id=17135841

Ah, but what about the tracking for ad targeting?

That tweet viewed by a million people isn't just a million tweet database reads, it's also a million tracking database writes.

I interviewed at a ride-share company (one that was not then working on self-driving cars mind you) once some 5 years ago or so and was shocked to find out that they had 1,000 software engineers. What did they need so many engineers for?! But then all their queries were hand-written to some key/value store w/o a query engine -- that has to explode your engineering needs a bit, though maybe still not to 1,000.

Distributed systems is the answer. Data isn’t in one sql database.

My take on this is that the client front end UI will load basic pages, but when it tries to load the feed there's somewhat Network error that causes the feed not to load.

There is a curious lack of data showing the impacts of these changes (though perhaps that's forthcoming as they roll it out at scale).

It should be obvious no one good stayed there. Either they were fired or left disgusted. You need to be really desperate to stick with Elmo.