Phantom Menance: memory leak that wasn't there
flakm.com
flakm.com
Kudos to this author for digging in - as a DevOps/SRE guy, I'd imagine this conversation often going in companies I have worked for like "Something is wrong with your dashboard" and my team being like "something is wrong with your application" and nothing gets done for months while managers point fingers and figure out whose problem it is.
This is not a great way to describe it. When a container is out of memory then the kernel ends it, and the kubelet is not involved. This is the main way that most users will experience OOM conditions. Note that it is not possible for RSS to exceed limit, because the task will end the instant it tries to realize memory that would have put it over the limit.
When the node is under total memory pressure then the kubelet uses working set to rank pods for eviction. Working set is used for that in an attempt to attribute kernel resources like page caches to each control group. But eviction due to node memory pressure should be rare.
> And then the hell broke loose. I received a message from our devops team that the application was scaling like crazy - we had a memory leak!
what sort of scaling are they referring to here