For the love of god stop using CPU limits in Kubernetes | Hacker News Reader

For the love of god stop using CPU limits in Kubernetes

github.com

41 pointsiljanevo43 comments

While I agree that CPU limits tend to make your performance worse, I don't think the delivery of the post is all too convincing (and is pretty heavy on the LLM-isms that it's putting me off from reading).

It mentions that a cpu request is a guarantee, but how is that enforced? If I have 32 pods running on a 32 core machine, each with 1cpu requested, what stops one of those pods using an unfair share? I assume we just rely on the Linux scheduler. If I have 16 pods with 1cpu and 1 pod with 16cpu, does the Linux scheduler make sure to give the 16cpu pod more time? Or are we back to using cgroups.

Requests are guaranteed no matter what.

> If I have 16 pods with 1cpu and 1 pod with 16cpu, does the Linux scheduler make sure to give the 16cpu pod more time?

Yes if there's CPU pressure the 16cpu will get 16x more than 1cpu.

Each pod has weight. These are written to your cgroup (cpu.weight in CGroups V2). CFS is scheduling based on these weights.

There's a part about it: https://github.com/inevolin/k8s-cpu-limits-analyzed#3-withou...

> If I have 32 pods running on a 32 core machine, each with 1cpu requested, what stops one of those pods using an unfair share?

I don't use all this fancy stuff, but wouldn't you set that up as each of the 32 pods is limited to running on a specific core? No sense letting them each run on all cores, because all of the X per core will get too big.

This always sounds good on paper and this is very common lore, but then when you get into production escalations a very common problem is a lot of software depends on limits for autoconfiguration of thread pools and even several runtimes (go and java for example, at least .net is mentioned in the article), yes you can usually set them with a flag but people have to know this, communicate it, enforce it. Basically replace adhoc what limits is doing for you automatically configuration wise

So this just all assumes you have a setup where all teams communicate the necessary information perfectly.. what happens in practice is workloads degrade at edge cases because there are 256 threads running for a thread pool instead of 4.

Go is particularly hilarious because it will just go and create a thread for every hardware thread in your system (my home system has 192 hardware threads), just in case it needs to scale up. So tiny utilities, little network proxies and the like add up to thousands of threads + stacks etc. for very little load.

To be fair to Go, it is a very common "rule of thumb" for a lot of software that do parallelism to spawn as much thread as there are cores. A lot of library and software in various languages do this. This is used to be safe 99% percent of the time since almost all CPU had at most 16 cores/threads. So even on dual CPU platform it was not that big of a deal. Now that you can get ARM and x86 CPUs with hundreds of cores (although a lot of use-case for those is running VM which will only report a certain cpu count), this rule of thumb can cause issue. But most devs are not running on platform with hundreds of cores, and there is always the option to set the thread count, so I expect this pattern will linger on for a while.

HPA uses limits for scaling decisions

Just give me the prompt

This is the only thing you need to know: https://home.robusta.dev/assets/stop-using-cpu-limits-inline...

The article behind it explains it better, but a quick glance at the image above is all you need if you don't care too much about the explanation.

https://home.robusta.dev/blog/stop-using-cpu-limits

The timing is great because I was going through that section of the CKA this morning. The only question I have is how guaranteed is the top right section? It seems to rely on the Linux scheduler, could you not have cpu steal from other pods if every other pod is trying to over allocate?

Yes, utter slop. Hacker News could do with support for flagging and ignoring such submissions.

Remove CPU limits. Keep CPU requests. Keep memory limits.

How is this “news”?

It was already the case in 2018.

Also, no mention of the scheduler overhead. And the maintenance overhead is the worst.

Scheduler overhead for CPU limits? Are you talking about the Linux Scheduler or the k8s scheduler?

It's not as much an "overhead" as it will mess up your latency. Limits don't stop you from using available resources until you hit the relative allowance in a CFS window, so a 1 CPU limit on a 32 CPU machine at worst gives you 32 cores for 3.3ms every 100ms.

Many orgs are just now discovering K8s at scale. Seriously.

I’m not sure why that is but a large number of the F100s I contract with are suddenly deploying 4 times the number of containers they had before.

Could it be that they're using coding agents to develop applications they previously wouldn't spent time on developing? Like internal tools, or experimental builds.

Looks like a bad copy of: https://home.robusta.dev/blog/stop-using-cpu-limits

"For the Love of God, Stop Using CPU Limits on Kubernetes" literally same title

Kubernetes CPU limits can also cause memory issues and OOMKills. It looks like a memory leak, the usual fix is a bigger memory limit, but the actual cause is the CPU limit. Here's some simple proof: https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-...

AI wrote most of this

I would expect the impact of cpu limits to be different between K8s providers. I have only used memory limits. If I had a pod that tended to be very cpu intensive, I would schedule it on it's own node group.

If you want the slop-free inspiration (2022): https://home.robusta.dev/blog/stop-using-cpu-limits

(The title of this was also stolen for this HN post, although the GitHub repo makes no mention of it...)

Author of the original here. Thank you for the mention!

Oh wow, what a surprise! I learned a lot from that article, and often referenced and sent it to my colleagues on my old team. Thank _you_ for writing it!

Limits are what give consistency when your pod gets scheduled on nodes with different amounts of load.

Some workloads will also consume all the resources you hand them without being latency sensitive at all.

I've handled outages of CPU time available getting suddenly compressed (we were running pod priorities with staging/prod on one cluster and up to 70% spots in 2019) and then learning that some very important applications outgrew their original requests, gone unnoticed because limits were removed a year or so prior. You can fix this with monitoring/right-sizing tools, but that requires your org to not be dysfunctional, and my style of platform engineering usually has to account for the org being very dysfunctional.

tbh I've never known search a 'feature' as limits and requests coupled with health probes to cause more problems in production than anything else.....

For the love of god - care about other pods on the node, especially in a multi-tenant setup.

Sorry for the cheeky response.

CPU Limits have a place, you don't want a bad change for 1 deployment object affect all neighbors by taking all the CPU. You need to be able to constrain the blast radius. This doc gives me strong AI vibes. Setting CPU limits isn't free. You still need to care about how the programming language that you use discovers those limits, and correctly handles them. For e.g. if you spin up a 100 Java threads, but only have 1 cpu as the limit, that's bad design.

That's what CPU requests are for.

CPU requests are cgroup weights.

But they also affect scheduling, right? If you set them too high, you will waste resources.

Exactly on point. Shit happens, performance bugs appear, someone messes up Kafka config and it starts consuming from the beginning of the world, etc. Limiting CPU is a must. I could see maybe if someone has a super good monitoring + oncall response team, then letting things go loose for a bit is a lesser evil than working out limits, but still.

I don’t think you understand how CPU limits and the Linux CPU scheduler work. CPU limits don’t protect you from something taking all the CPU; that’s what CPU requests do. Limits throttle your pods even if the CPU is idle/free to do work.

From the docs another comment linked:

> The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.

So I guess limits _would_ protect other pods to a degree. Though I agree that it doesn't seem worth the tradeoff of your pod getting constantly interrupted while the rest of the box is sitting idle

That's literally not what the documentation says.

https://kubernetes.io/docs/concepts/configuration/manage-res...

Where do you think the contradiction is? A CPU limit of 1 does not prevent an application from using 8 cores worth of compute at once, it just limits the compute to 1 core per 100ms slice on average, and usually that means getting unscheduled for 7/8 of the slice, if the app is using all available resources and nothing else contests them (weighted by requests).

According to the documentation, CPU requests don't cap usage. They're not used for that. They're used to gauge how much CPU you say you will use. They're used for allocation and pod assignment. They never throttle you. They never cap you. According to the documentation, CPU limits are the only mechanism to prevent exceeding your resource allocation. You directly said the opposite of that. If that's not what they actually do, then FIX. THE. FUCKING. DOCUMENTATION. Either the documentation is wrong or incomplete, or you are wrong or incomplete.

That's the contradiction. And, to be clear, I think you're both wrong and incomplete.

It's partially a problem that how it's being documented is either really misleading or fundamentally incomplete. That may be because that's just not how Google imagined Kubernates was going to be used. If you need to understand the CPU scheduler to be able to use this option in the first place, then the documentation should explain that directly or by referring to more information elsewhere.

I don't see the contradiction to what I said or what the prior commenter said.

CPU requests still protect you from something using all the resources, because the weights correspond to your number vs all the other pods on that node, and the scheduler will not schedule pods so they exceed the total available resources.

So more accurately, requests prevent exceeding limits under contestion, and limits always do.

Hahaha! Thanks for the laugh.

Clearly written by a developer (who asked ChatGPT to generate a page like this) and not a sysadmin.

It's the classic "developer can't optimize their stuff, so they ask for infinite resources to cover their mistakes" farce.

No, keep the CPU limits. They exist for a reason. Fix your app.

Hilarious, another kubernetes footgun - the gift that keeps on giving.

One of my former colleagues wrote this on how Uber approached the same CPU-quota throttling problem, but with dedicated CPUs as the solution: https://www.uber.com/dk/en/blog/avoiding-cpu-throttling-in-a...

Essentially, they avoided CFS quota throttling by assigning exclusive CPU cores via cpusets. That sacrifices some burstability and packing efficiency in exchange for stronger, more predictable CPU isolation.

Reply on news.ycombinator.com