Backend of Meta Threads is built with Python 3.10
twitter.com
twitter.com
Some worthwhile references:
[1] https://engineering.fb.com/2016/08/31/core-data/myrocks-a-sp...
[2] https://research.facebook.com/file/529018501538081/hhvm-jit-...
[3] https://research.facebook.com/file/700800348487709/HHVM_ICPE...
But… how? Considering all the transactions in flight, and everything? And did you ever Test disaster recovery with that setup?
I’ve worked on relatively big projects, but FAANG engineering is like some entirely different field of software engineering. Fascinating.
It was a lifetime ago I ever did DB administration (postgres in my case), but the write-ahead-logs being replicated out independently was extremely important for point-in-time recovery, such that you could always take the latest backup, zip forward through the WAL, and recover to any arbitrary point in time you want, so long as the WALs were available. I wonder how much something like this would have been done at FB scale.
> Considering all the transactions in flight, and everything?
If I remember, we used --flush-logs --master-data=2 --single-transaction, giving it a consistent point-in-time dump of the schemas, with a recorded starting point for binlog replays, enabling point-in-time and up-to-the-minute restores. Nowadays you have GTIDs so these flags are obsolete (except --single-transaction).
--single-transaction does put extra load on the database—I think it was undo logs? it's been a minute—which caused some hair-pulling, and I believe they eventually moved to xtrabackup, before RocksDB. But binary backups were too space-hungry when I was there, so we made it work.
Another unexpected advantage of mysqldump vs. xtrabackup, besides size, was when a disk error caused silent data corruption on the underlying file system. Mysqldump often read from InnoDB's buffer cache, which still had the good data in it. Or if the bad block was paged back in, it wouldn't have a valid structure and mysqld would panic, so we knew we had to restore.
> And did you ever Test disaster recovery with that setup?
Yes! I wrote the first version of ORC. This blog post is from long after I left, but it's a good summary of how it worked: https://engineering.fb.com/2016/10/28/data-infrastructure/co...
It wasn't the best code (sorry Divij)—the main thing I'm proud of was the recursive acronym and the silly Warcraft theme. But it did the job.
Two things I remember about developing ORC:
1) The first version was an utter disaster. I was just learning Python, and I hit the global interpreter lock super hard, type-error crashes everywhere, etc. I ended up abandoning the project and restarting it a few months later, which became ORC. In the interim I did a few other Python projects and got somewhat better.
2) Another blocker the first version had was that the clients updated their status by doing SELECT...FOR UPDATE to a central table, en masse, which turns out to be really bad practice with MySQL. The database got lock-jammed, and I remember Domas Mituzas walking over to my desk demanding to know what I was doing. Hey, I never said I was a DBA! Anyway, that's why ORC ended up having the Warchief/Peon model—the Peons would push their status to the Warchief (or be polled, I forgot), so there was only a single writer to that table, requiring no hare-brained locking.
Also, Facebook used to do ahead of time compilation (HPHPc) but eventually HHVM managed to outperform it.
I'd think the same or bigger performance benefits can be reaped if you just start with stuff like OCaml / Haskell / Golang / Rust from the get go.
Arguably picking almost any other tech would have worked as well because they would have doubled down on it as well and made sure that all the pieces that they need are in place and are working.
Thanks for elaborating, though I definitely don't share your conclusion.
Have they really done this for OLTP workloads running in mysql? I know they do it for OLAP though.
Asking because I’m curious about these early stage mega-tech companies that are working at a scale I can’t fathom.
No idea if this was a credible leak of actual 2007-ish FB php, but it doesn't look like what you're describing.
Facebook was founded in 2004. Whatever fancy tech you're thinking that would fit the round hole, it didn't exist then. They also never thought it would serve a couple billion of users one day, so they built it in whatever they had and knew at the time.
Same for Instagram and Python (they started with Django).
"It's running on Instagram's #Cinder fork that includes a JIT, lazy-loaded modules, pre-compiled static modules, and a bunch of other interesting tweaks against vanilla Python 3.10"
From the Tweet.
Personally, I don't care what language people use. We all know that if you have money, you can just throw in more servers. Until your CFO and CTO decided that the next big thing is to do "Cost Saving" (in the eye of recession).
I'm surprised in 2023 people are still debating mainstream programming language "PROD" readiness.
It's like a "cool kid" competition.
This is a huge loss for the world, and makes me a bit angry.
Edit: I had falsely assumed that because GP missed out on the JITing etc. described in the linked tweet, that Twitter was still inaccessible to the open web. However, trying again now, Twitter is again viewable without an account. So Twitter's communication about needing an account to view it was lacking, but so was my diligence.
Yesterday, I read an article about Tour de France stage and it linked to about 4 tweets with videos which are not playable without an account.
I just hope this gets noticed by mainstream media and they change their practices instead of assuming that everyone has a Twitter account.
Definitely not true at Facebook scale. You need to be smart, fast, _and_ throw more servers at it.
We both know the issue has always been how you store the data. Be it your main data storage, or the need of multiple storages for different use cases, etc.
The app server itself can be implemented in any languages. It's just a data orchestration + transformation logic anyway...
Being able to handle spikes and iterate quickly is probably more important.
In other words, Facebook felt compelled to write a compiler when they had hundreds of millions daily active users.
It's not the end of the world because they have deep pockets and a problem domain where preserving the interface (I.e., keep everything still running on PHP) and optimizing the infrastructure that provides it is more cost effective and least disruptive than switching to a tech stack that is more performant.
Sometimes that actually meant rewriting the thing from scratch, and other times that meant adding something else as a middleware to prevent touching the original thing.
For the most public success story, React was a complete rewrite of the thing before it, for example.
It's mostly hyperbole but there's definitely a kernel of truth to it. I generally dislike rewrites and I wonder if FBs approach influenced me more than I realized at the time.
So they created their own language Hack so allow a slow transfer from PHP to Hack. Which is basically the idea behind Kotlin.
Why would a massively successful company toss a stick in their own spokes by tearing the product to ground?
I'm also very skeptical about big bang rewrites, but there are points where you need to migrate off certain tech. Ideally you can make that transition piece by piece though, but that also introduces its own set of problems (now you have two systems and the new one has to inherit some of the baggage of the old one in order to be compatible).
Those systems are well understood and well tested, so it's not cost effective at the moment to embark on a complete rewrite. Current AI coding systems are also very unreliable, so it's just a matter of time before those two meet on the graph and voila, another boom in moving the legacy world into safe languages.
Facebook was already at a scale larger than 99.99999 percent of sites before they had HHVM
Not only that, the community felt the pressure to write several JIT compilers already.
So....
Ruby is the exception to the rule.
Friends don't let friends start new Ruby projects in 2023.
Yea actually. Bash/CGI could handle plenty. If thats your most comfortable language then go for it.
About 80% of the code I now write is in bash, because usually the problem ones trying to solve has a trivial solution if you just use existing tooling in creative ways.
At one point I wake up two hours later and realize I could have finished this in 15 minutes with Golang or even Ruby.
Ultimately these these choices are all about trade-offs. Maybe python is fine for them, maybe they've built themselves into a corner. Time will tell.
And I have definitely seen projects fail due - in part - to language choice. Of course projects can succeed in almost any language but that doesn't mean the language choice is irrelevant.
Just Google "python vs c++ speed" and you will find hundreds of examples (or "python vs rust speed" - Rust is essentially the same speed as C++).
Here's the first result - they got a 25x speedup:
https://towardsdatascience.com/how-fast-is-c-compared-to-pyt...
Plus, instruction and cycle counting is low hanging fruit compared to memory latency. You can cache-optimize a program in any language so long as the memory representation of some data is relatively transparent.
That's a very valid question but in the case of Facebook they already have them so why not use them for that?
I mean yeah, they made their choice -- use HHVM and it likely served them very well. I am just pointing out that in their case sourcing extra (or even any at all) C++ devs is a non-issue because they already have plenty.
Fully agreed with your memory latency remark.
Technically choosing C++ over Python will save you several orders of magnitude more than nanoseconds.
Though C++ might be a bad example. I'd replace Python web app with Elixir or Golang.
the claim is usually more about how the maintenance costs are just way higher versus a statically typed language
https://www.techempower.com/benchmarks/#section=data-r21&l=z...
That’s off by multiple orders of magnitude, if you believe their reports of 30 million sign ups in under a day.
https://www.theverge.com/2023/7/6/23786108/threads-internal-...
But it’s not “just Python”.
The OS state that needed to be loaded is not Python.
The network gear firmware is not just Python.
The databases are not Python.
Where it becomes just Metas Python is after a whole lot of other necessary work is done by other smarter people (the ones who master the physics of building the machines), not just some dweebs who got pulled the internet onto the host.
These articles are just primate rage bait
Absolutely shocking.
Why would they use Django? I did some small projects in it but I assumed it wasn't very fast and wouldn't be suitable for a big app like this. I would like to know the pros and cons. Why didn't they build up something in C++ or Rust? Won't python limit the speed of responses despite being build the hard stuff in compiled languages? Sorry for being so naive, I am an amateur.
(Heavily modded, run on a custom Python JIT, and using an extremely custom FB-developed database, also used for IG and FB.)
It's Django because IG was originally written in Django back in the day. FB's general approach to scaling is keep the interface generally similar, and slowly replace things behind the scenes as necessary — rather than doing big rewrites. It's seemed to work pretty well.
Ultimately the language used for the web server isn't a huge deal compared to the performance of the database, for these kinds of massive social apps. Plus, well, they do have the custom JIT — but that's very new, and when I first joined IG in 2019, we were running vanilla Python in production.
Thanks also to every other answer I received for my question, I appreciated all of them.
Why Django? Because that's what it originally was. Same with YouTube frontend btw. The apps just grew and grew.
If so, do you have any recommendations or suggestions for someone getting their feet wet on this?
Some of my team mates work on driving internal adoption of Cinder, rolling out new versions of Python everywhere, improvements to our build and packaging systems, supporting other Python tooling teams, etc. There's a lot of cross-functional project work, and our primary goal is to improve the Python developer experience, both internally and externally wherever we can.
Is this just a Flask problem, or does Django have the same issues?
I know django doesn't have that "shiny factor" to it these days - but it's very reliable.
> mixed messaging on best practices for scalable apps
The WSGI stuff can be kinda confusing and is used across a lot of python frameworks including django and I think flask?
My advice for "simple scaling" is to start with a separate Postgres instance, and then use gunicorn. Use celery immediately if you have any "long lived" tasks such as email. If you containerize your web layer, you'll be able to easily scale with that.
Finally - use redis caching, and most importantly - put Nginx in front! DO NOT serve static content with django!
> the ecosystem feels overbloated with vapor ware extensions.
This still exists to some degree for some more niche stuff, largely because of it's age. Although impressively they'll generally still work or work with minimal modifications. It's popular enough and old enough that most normal things you'd want to do have decent extensions or built in support already.
I've mostly used Node, Java, and Rust, with about 12yrs experience now. Only about one year with Python and Django recently. I am so much more productive than anything else I've tried. Using anything else feels like a mistake for web apps or CRUD apis. I also use type hints.
I use Django + django-ninja + django-unicorn for dynamic UIs.
Building sidewaysdata.com with django right now! Govscent is in Django too, the ORM is nice to combine with AI stuff already in python.
I'm also following the Ash framework which looks promising.
If you want to purely make APIs though, give FastAPI a try.
The ORM and django admin are killer features out of the box that the other frameworks don't have. I will say though that FastAPI is really nice, especially if you need async support. However, I have found that using django ninja [1] adds a lot of nice to haves that FastAPI has to django that makes it much more fun to use again.
There are some footguns around N+1 queries, but they are general to all database interfaces and pretty easy to avoid IMO.
It's a different strokes for different folks sort of deal.
Django is very opinionated, in a way that is "eventually" correct (i.e. they might have had some bad opinions many years ago, but they have generally drifted in a better direction). If the opinions don't align with what you're building, the escape hatches are not generally well-documented or without penalty.
Flask is minimalist and flexible. Once you find a "groove" to building with it that fits your sensibilities, it's quite, quite nice. That being said, the most recent versions of flask have excellent documentation, imo, and the tutorial is a bit more opinionated in a highly productive way. The "patterns" section of the docs is also super useful for productionizing your app.
Personally, I prefer the combo of Flask+SQLAlchemy, and eventually Alembic once you decide that's good for your app's maturity level. I respect Django a lot, I just enjoy the explicitly "less magical" aspects of a Flask stack, which is an opinion-based trade-off imo.
Some will use `python manage.py runserver` in production, and they are using the defacto wrong set up. Don't ever do that.
It’s millions of lines of code, you can’t just change it to be Java one day.
It's probably hurried along python code and fairly simple horizontal scaling. Some population of individual instances are probably pegged, and random requests are frequently landing on their backlogged request queues.
This seems to have been rushed out the door to capture a unique market opportunity, and they'll clean up the engineering once they get engagement.
So not entirely just python 3.1.
Great work, much appreciated.
Python doesn't go by semver rules. Semver isn't a definition for all software project versioning systems everywhere.
Get it? Threads, threading...? OK, I'll see myself out.
In fact a Thread thread has already been on the front of HN.
So until it either solves its privacy issues of becomes part of the fediverse, audience is crippled.
Edit: Someone kindly pointed out that you can see the content in the browser without an account from following a direct link. (You just can't have the app installed or it will open that and drop you on the login page.) This makes me a lot happier with the product!
Based on that, I think it's safe to assume that a full featured web version of Threads should be up within a year. Threads already has some web presence (you can view threads and profiles).
Instagram can be used in a desktop browser just fine.
Its not open to the web yet. Supposedly, it will be
However, I have a much larger following on Twitter, so it was the tweet that got picked up by someone here on HN.
We'll see how this looks like in twelve months.
It’s still perfectly acceptable for Twitter users to discus other platforms on Twitter.
Wordpress wouldn't be this successful if this approach didn't work. After all, without caching it will happily take over 20 seconds to render the main page.
Is Cinder something that could help optimize real-time streaming? We had a UDP stream and then through multiple gstreamer and nvidia deepstream magic (which I believe the senior dev implemented in Python) we perform some ML inference on the stream in real-time.
However, latency is a major issue here and to get to our MVP we didn't really prioritize optimization, as is tradition.
So now I'm wondering if Cinder as something that can be used to optimize real-time data streaming is a thing or whether me asking this just shows I don't understand its use case.
Either way, thank you advance for your insight.
(Also we used Django which I am now wondering if I should have switched out for FastAPI, but that's a separate question)
Would you recommend using Cinder for stack made of - Django - Cython - Numpy
Thanks!
For example: you start a main process, warm it up with a few requests, run the JIT compiler and then fork off worker processes to handle the main chunk of traffic.
As of now, it requires hand-tuning to get the best possible performance.
In terms of use cases, Cinder does the best when faced with "business logic" code (lots of inheritance, attribute lookups, method calls, etc). It can speed up numerical computations too, but you're probably better off using a library if that's the majority of the workload.
Because it's adding well over a million new users each hour which is pretty staggering.
But still there probably is all sorts of juicy details that would be neat to hear about
But I think on net it was still an easier problem than scaling a startup product from zero, because the tooling and processes to support huge usage is already so mature at these big companies.
A few million new people is less than 0.1% of the total. It's entirely within expected variance.
My naive assumption is that they have some neat simulated user/load system to help identify any scaling issues that might arise when launching a service, and the extreme % growth. That 3.8 billion was spread over 15 years.
Surely it makes massive difference to hardware utilization?
I like Python and javascript but there’s perfectly fine tools around now for building systems that allow one server to do a lot more.
They say you should not prematurely optimize. Equally you should not be prematurely unoptimised.
It's better, but not the limiting factor.
I’d use the analogy of a Japanese bullet train as the compiled language and a diesel train as Python.
Python is my primary language along with JS but if I was building a new service for Meta I wouldn’t use either at the back end.
Python is likely faster to push out an app like this than say rust or even go.
Also why spend time optimising something that will be dead in a few months?
Instagram stories crushed Snap’s DAU, Reels is gobbling up TikTok user-minutes…bleeding other services is what Meta excels at
It loads a javascript-driven animation of what appears to be a galaxy, any not much else. There is a QR Code at the bottom right corner that appears to be invalid.
Not sure what I'm missing. 10MM+ signups through what mechanism? There is no signup form as far as I can tell. Do they restrict full content access for linux-hosted browsers?
edit: after saving and reversing saturation, I've resolved the QR Code to https://www.threads.net/download/redirect, which only offers binary download options, and only for mobile platforms.
Where is the web interface?
But that's really odd because most mobile MVPs are either just PWAs or some React Native shit compiled to binaries.
Why the hell wouldn't they just release it on the web?
I'm a data engineer trying to teach myself React so I can build a full stack app using some APIs and it's certainly a good exercise, but I could also certainly build that in Python. Your app looks great and I'd like to explore that more!
P.S - fellow runner and cyclist and out and I'm checking out the app more now.
Its an interesting dynamic: popularity draws attention, attention helps adress pain points, this makes the platform more attractive and the popularity increases...
No point in supporting that user-hostile tire fire.
This could have been on Mastodon.
Geez, it is you who is the authoritarian for coming so hard at the person who asked a pertinent question.
Geez.
Don't you find it ironic that you are proposing we do on HN exactly what Twitter did, which is probably why you suggest not linking to Twitter?
I would say that the amount of tweaks that prop up Python is no cause for celebration.
Too weak for someones hobby project, strong enough for global scale.
The same is true here. They spent so much money getting Instagram to scale. They created Cinder to try and fold their hacks/tweaks back into Python. Spinning up some Rust/Java/Nim/Zig/Pony/Whatever stack sounds like it'd be fun... but spinning up a stack with which you're already VERY familiar, sounds like a money-maker.
I'm totally agreeing btw, I realize this came off like it might be a counterpoint.
My personal suspicion is that the secret sauce is simply to have a sufficient budget to hire an army of code quality engineers. I could be wrong, though.