Comment by toast0 | Hacker News Reader

Comment by toast0 | Hacker News Reader

Parent Full thread

File descriptors are a limited resource, but the limits are huge. My little 2GB instances on GCP claim a limit of 1M; FreeBSD autotunes my 16GB servers to 0.5M (but I could increase it if I needed).

I just don't know how you have a 1000 node system and you can't manage to heartbeat everything 2x a second; I don't think you need that many heartbeats in most systems, but it's just not that much work. The only way I can see it being a lot of work is if your nodes are very small; but do you really need a 1000 node esp8266 cluster and you can't get anything bigger for the management node?

> And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too.

How big is your heartbeat, is the response really going to be in multiple packets? With a 500ms heartbeat and a typical 3-5x no response => dead, you're going to hold onto the partial heartbeat for like 2 seconds.

> File descriptors are a limited resource, but the limits are huge.

Maybe in the linuxy servery context we're talking about, but I'll note I've encountered EMFILE and ENFILE errors attempting to open a mere couple hundred files on iOS simultaniously. And while my own experience is over a decade old, here's someone hitting those limits this year: https://github.com/bevyengine/bevy/pull/17377

(Context for my own encounter: gamedev, with massively parallel asset streaming and multiple layers of direct blocking dependencies in deserialization. One of my hackarounds involved temporarily closing the least recently read files, and then reopening them on demand in our VFS abstraction. Of course, for non-dev builds, you can just use a pack file and `pread` a single fd from multiple threads, but loose files were handy for incremental dev updates...)

> The only way I can see it being a lot of work is if your nodes are very small

Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often.

> you're going to hold onto the partial heartbeat for like 2 seconds

And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats.

Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.

For Linux, the limit on FDs is just memory.

Reply on news.ycombinator.com