The performance difference between RAM and disk is not an acceptable tradeoff. RAM will be tightly provisioned and jobs will be killed rather than letting the machine OOM.
I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine?
I personally do have swap on all my production machines. I’m not relying on it, but it acts as a safety net in case unexpected memory usage sneak in some jobs. This goes with a sane periodic review of host metrics of course to ensure safety nets do not become the norm.
I much prefer a slightly delayed job than a failed job.
We prefer to have job OOM kill and get retried elsewhere (which could be a completely different machine) and we have plenty of infrastructure that makes this trivial. This infrastructure also deals with other types of partial failure, such as complete machine failure.
As mentioned above, paging introduces strange pref behaviour. Which may not always be important, but if your working under tight latency requirements then paging can push you over that boundary.
That sounds strange when we’re happy to see jobs die entirely (that screws latency). But the issue with paging is you have no idea when it’s gonna hit you, and may impact a job that’s behaving perfectly fine, except something got paged out by a badly behaving job.
Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour, and making cause-and-effect highly correlated (oh look thing X just OOMed, probably means thing X consumed too much memory. Rather than thing Y has strange tail latency because thing X keeps consuming too much memory). It’s failing fast, but for memory rather than exceptions.
- paging incurs seemingly random performance degradation of processes and should be avoided
- if you have a form of task queue/job distribution system which handles automatic re-run, and can afford at no business cost to restart a process from scratch, then disabling swapping allows fail fast behaviour
- otherwise swapping can be used as a safety net for programs that would be better off slightly late than restarted from scratch
- both scenarios require sane monitoring of process behaviours, to catch symptomatic failures/restart in case 1) and recurring swap usage in case 2)
Even if you're not under tight requirements, swap can do strange things. I've actually seen situations where hitting swap, even trivially, can cause massive increases in latency.
I'm talking about jobs which took 10s of milliseconds to complete now taking multiple 10s of seconds.
I've even seem some absurdly bad memory management where Linux will make very very poor choices about what to page out.
> Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour
1000% agreed. Fail fast rather than fail slowly.
> I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine?
I expect the commenter means whatever thing kicked off this job will detect it failed and do something about it (try it again elsewhere, log an alert triggering an admin to go provision more capacity, etc.)
Their view is that a failure is easier to troubleshoot and fix than success with intermittently anomalous characteristics.
To be honest, what I really wanted here is to force the commenter on agreeing that whatever the architecture/framework/etc., there is no magical solution, and saying that it will "be compensated-for at a higher level" just hides the only 2 real possibilities:
- Kill the process and restart it somewhere else (no swap)
- Wait for the process to finish anyway (using swap)
Both choices have their own pros and cons, as discussed in other comments. But arguments such as "it's better to have predictable process termination", and "under tight latency requirements then paging can push you over that boundary" are exposed as much less meaningful once the problem is stated in simple terms.
If the process indeed uses more memory than planned, then you have no guarantee than restarting it will work this time. Worst case, you can even have to allocate urgently a new node to handle that 1MB of memory above what was available. Not sure that is really the best solution "under tight latency requirements".
My experience would indeed be the reverse: if you are faced with unexpected memory usage growth, then having some swap on the side to allow your process to finish makes for a much smoother production system than having to restart it all from scratch somewhere else/at an other time.
In my experience, production is not the place where you want to enforce ideological concepts such as "a process shall not use more memory than what was planned". Production is the place where you make compromises and account for everything that slipped through these concepts, because you have no choice: it _has_ to run. Staging would be a good place for this though, as it's much less critical if something fails, and you have more time to fix and account for it.
I really don't want to debug/profile/release/deploy a program in a hurry just because somehow it ended up using more memory than planned. (or convince infrastructure to deploy a new node type ASAP to handle that new memory constraint).
Now one could argue that such unplanned memory consumption should have been caught earlier in the development process, to which I would reply:
- Not everyone is able to properly test programs in a production like environment, especially when dealing with resource-hungry processes. In our case for instance a typical staging machine would be a downscaled version of the production one (less cores, less memory, less bandwidth).
- I tend to plan _also_ for what should _not_ happen (because it will happen). Even with a perfect environment to calibrate programs before a release, there will be releases in which the memory usage was not accounted for. Guaranteed.
Gray failures (like process slowdowns), on the other hand, are fairly difficult to design for, and detect. And it can wreak havoc on distributed systems if one of the nodes suffers a gray failure.
Sure, swap is bad on steady state loads. But for transient loads it works
Unless you like playing "why did my service got killed this time" every once in a while
There is no playing: the error is logged and quite clear, by the application or by the kernel.
Being consistent, predictable and as deterministic as possible is a key feature of reliable software, and paging into swap kills all that.
Paging into swap often has almost no perceptible effect in regular usage unless you are running software with microsecond guarantees on infrequently accessed processes (those that get swapped out).
Kubernetes as a platform recommends disabling swap completely, and you have to explicitly allow nodes to have a swap, otherwise it fails.
This is sane behaviour if you're dealing with a large cluster with a complex architecture you no single person could or should know all the ins and outs of. There is no "let's log on to the machine and see what's happening" when dealing with this types of architectures, even at smaller scale.
And a massive part of Go's target is exactly these modern architectures/workloads...
But still, you could end up with a case where your pods start using a little bit more memory (for a while) and getting killed by OOM