Hacker News Reader

Based on the descriptions of the tasks here, they're going to run out of CPU for the actual processing tasks long before they run out of CPU for the actor management. So it doesn't matter whether their actor implementation is capable of spawning 25 million actors doing nothing on commodity hardware when they can probably only keep maybe 2 times or 3 times the number of CPU's worth of actors busy anyhow.

View on HN

This is indeed correct. Yes, we have comparatively few actors. One thing that is not explained in the blog post however, is that we can run several indexing pipeline on the same machine. We have a bunch of tricks in our framework to run ~1000ish index pipeline on the same server. That's a target of around 10k actors in the most extreme case.

We also send very few messages: the smallest message we send is represent batches of log lines.

As a result, our framework does not really care about performances. On one thread, it can process ~2.5 millions messages on my laptop: this is well above our needs, but it is correct to say that our framework is very slow compared to others.

Reply on news.ycombinator.com