Hacker News Reader

Comment by est31 | Hacker News Reader

Nice, I laughed out loud when I read these two paragraphs:

> Initially, we tried messing with some garbage collector parameters we didn’t really understand, but to our surprise that didn’t magically solve our problems so instead we disabled garbage collection altogether. This increased our memory usage, but our automatic on-demand scaler handled this for us, as the graph below shows

> Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.

View on HN

In case anybody missed this: https://instagram-engineering.com/dismissing-python-garbage-...

Yes, but the Python garbage collector is only needed to handle cyclic references. Objects whose reference count drops to zero are still freed even with GC turned off. So AWS didn't make a mint from the experiment.

I would also expect Instagram to be on Facebook's infrastructure these days ;)

Right. The article describes them saving resources in the end from it, but I guess that was too reasonable for satire.

I'm sure the cost of getting all of the devs to write Python that safely runs without GC is far les than 10% of your AWS bill for Python services... Maybe at Instagram it is, but it won't be at 99% of companies.

It isn't like most Python devs you interview are going to have any kind of a reasonable answer for "How would you write XYZ - with the GC disabled?" I'd bet the ones with a good answer cost >10% more on average.

So swift using reference counting instead of a GC was BS marketing where the python approach (using both where it make sense) is a precedent and a superior solution?

FWIW, there's actually a CPython patch from Instagram [0] that proposes "Immortal Instances" (and I understand that YouTube had their own set of patches for "Eternal Refcounts") which aims to alleviate some of those issues and reduce Copy-on-Writes.

[0] https://bugs.python.org/issue40255

Actually, I think this was meant as a reference to Twitch's experiences with the Go garbage collector:

https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

or may be: https://blog.discord.com/why-discord-is-switching-from-go-to...

The Discord Go to Rust seems quite sensible engineering.

Sensible? I guess when you dig yourself into a hole, wanting to move to a shallower hole seems sensible. It showed that they understood neither their requirements nor their chosen tools, not surprising given that their stack is Elixir, Go and Rust, languages that there aren't too many experts for, and such consistent choices are a clear display of inexperience. There's just no way that technical sensibility is a chief concern of a company that goes from Elixir to Go to Rust within a few years.

If they had started with a runtime with better GCs or chosen, say, C/C++ from the beginning, none of this would have been necessary. They clearly choose hype first and everything else second. If that's their technical evaluation process, they are likely to choose wrong the first time, the second time, the third time, and however many times they need to rethink their choices, and every time their rationale would seem sensible when it becomes clear that their previous, ill-advised, choice is wrong, and before their new, untested choice shows its own problems. Making a bad choice and then wishing to fix it by making another using the same process that's led to the first doesn't inspire confidence.

Yes, if they used C/C++ from the beginning, they wouldn't have run into GC issues, but they would run into another whole set of issues exclusive to C/C++ (free-after-use, buffer overflows, annoying threading bugs, etc.)

Just like they've run into issues with Elixir and are going to run into plenty of serious issues with Rust. The belief that your chosen language/runtime does not have some very serious issues is the mark of inexperience. The only thing you can do is know what they are and what you can live with. Choosing a hyped language makes the first part very hard; their history shows that they consistently fail on the second part, too. It looks like it's a very inexperienced team, which tech has plenty of, but turning their blunders into success stories is precisely the kind of amateurism dressed in bluster that this post is rightly ridiculing.

There's an argument that it's good to avoid premature optimization - build your thing with something easy like rails or elixir and then if you run into performance issues rewrite those bits in something faster.

It's not about premature optimization. They clearly didn't know enough about Go when they decided to use it -- partly because few people do, but they did know that -- and they clearly don't know enough about Rust, either. They're stumbling from one hype to another based on sales brochures and bragging about it.

Again, Tech is full of such low-experience teams writing this kind of unimportant software and rewriting it every couple of years, and I guess it's good that someone volunteers as a lab rat for new languages (even if they do it over and over). But they're so inexperienced that they can't even see they're being snooty about their ignorance of their own requirements and tech.

> free-after-use

I wish ;)

Oops, meant to say "use-after-free"

Elixir could be argued as almost-sensible if they made the typical chat product backend decision to bolt some features onto ejabberd and pretend like they're hot shit at designing distributed servers. Though I would still stick with vanilla erlang because if you use elixir without also knowing erlang, you'll find yourself up shit creek without a paddle eventually. But a halfhearted google doesn't even confirm that they built on ejabberd.

I work for twitch and I've seen that article before and it doesn't seem to fit the pattern of we messed with GC and it didn't fix out problem. The solution in the article uses weird features and behaviors of the golang GC and how VMs allocate memory but it did work very well for them.

Can you please remove the channel points button for me? Thanks

See also Hotmail when they started migrating it to Microsoft tech.

The servers were stable for a day or two, so they cooked up a reaper process that would sequentially reboot every machine in the cluster, wait for it to start routing traffic, then reboot the next one. Every machine got at least one reboot a day from this.

The moral authority with which I used to share that anecdote has eroded quite a bit in the years of Continuous Deployment. Most of my endpoints reboot at least once a week, except for holidays where we occasionally discover longevity bugs. With autoscaling some instances will be up for hours at a time (and if we are not very careful we'll completely pollute our server statistics).

One advantage of CGI is that the worker process terminates, and all the resources get freed.

Not that I recommend CGI for anything but very low usage sites, typically internal ones with a fixed user base.

Not to worry. We'll cloud-enable fastcgi and call it Serverless.

eBay did the scheduled reboot thing around 2000, but on a massive scale.

Their front-end servers were Windows ISAPI, and leaked memory like a sieve. So there was a documented process for rebooting groups of servers with a diagram dividing the front-end servers into rebootable clusters and the time between reboot intervals.

And yes, docker/k8s sure reminds me of those days when reviewing logs today and seeing some pods with unexplained non-zero restart counts.

Source: worked there, read the documentation.

> Fun fact: As our spreadsheet met strong statistical guarantees of randomness, we were able to reuse it to replace our application’s CSPRNG.

Made my day.

My favorite

> Tomorrow we will try to understand our $43,398,207.34 bill for April's AWS usage.

And after understanding it, nobody will be dumb enough to volunteer to delete a third of the billed resources.

They will just have a program do the deletion randomly and call it chaos engineering.

There has to be at least 6-7 good blog posts to come out of that, too.

That's an interesting idea!

Would work for EC2, not so well for resources you don't alert on like S3 buckets, etc.

Reply on news.ycombinator.com