Comment by dilyevsky | Hacker News Reader

Comment by dilyevsky | Hacker News Reader

Parent Full thread

This is subtly incorrect - as far as Docker is concerned CFS cgroup extension has several knobs to tune - cfs_quota_us, cfs_period_us (typical default is 100ms not a second) and shares. When you set shares you get weighted proportional scheduling (but only when there's contention). The former two enforce strict quota. Don't use Docker's --cpu flag and instead use --cpu-shares to avoid (mostly useless) quota enforcement.

From Linux docs:

  - cpu.shares: The weight of each group living in the same hierarchy, that
    translates into the amount of CPU it is expected to get. Upon cgroup creation,
    each group gets assigned a default of 1024. The percentage of CPU assigned to
    the cgroup is the value of shares divided by the sum of all shares in all
    cgroups in the same level.
  - cpu.cfs_period_us: The duration in microseconds of each scheduler period, for
    bandwidth decisions. This defaults to 100000us or 100ms. Larger periods will
    improve throughput at the expense of latency, since the scheduler will be able
    to sustain a cpu-bound workload for longer. The opposite of true for smaller
    periods. Note that this only affects non-RT tasks that are scheduled by the
    CFS scheduler.
  - cpu.cfs_quota_us: The maximum time in microseconds during each cfs_period_us
    in for the current group will be allowed to run. For instance, if it is set to
    half of cpu_period_us, the cgroup will only be able to peak run for 50 % of
    the time. One should note that this represents aggregate time over all CPUs
    in the system. Therefore, in order to allow full usage of two CPUs, for
    instance, one should set this value to twice the value of cfs_period_us.

> "Don't use Docker's --cpu flag and instead use"

This is rather strong language without any real qualifiers. It is definitely not "mostly useless". Shares and quotas are for different use-cases, that's all. Understand your use-case and choose accordingly.

It doesn’t make any sense to me why —cpu flag is tweaking quota and not shares since quota is useful in tiny minority of usecases. A lot of people waste a ton of time debugging weird latency issues as a result of this decision

With shares you're going to experience worse latency if all the containers on the system size their thread pool to the maximum that's available during idle periods and then constantly context-switch due to oversubscription under load. With quotas you can do fixed resource allocation and the runtimes (not Go apparently) can fit themselves into that and not try to service more requests than they can currently execute given those resources.

And how is that different from worse latency due to cpu throttling from your users’ perspective?

this is a predictable worse latency due to CPU throttling, i.e nothing is suddenly introduced in the system. but the other case is worse i.e a non critical microservice can cause outage on your critical microservice.

imagine some non critical system like blog-service suddenly causing 2-3% new order creation failures

Fixed queue, so it'll only take as many as it can process and reject the rest, which can be used to do scaling, if you have a cluster. With shares it would think it has all the CPU cores available and oversize the queue.

Doesn’t answer my q really. At least in kubernetes scaling is done by measuring usage against the request (shares) not the limit (quota)

These two options are not mutually exclusive.

When you want to limit the max CPU time available to a container use quotas (--cpus). When you want to set relative priorities (compared to other containers/processes), use shares.

These two options can be combined, it all depends on what you need.

> Don't use Docker's --cpu flag and instead use --cpu-shares to avoid (mostly useless) quota enforcement.

One caveat is that an application can detect when --cpu is used as I think it's using cpuset. When quota are used it cannot detect and more threads than necessary will likely be spawned

It is not using cpuset (there is a separate flag for this). --cpus tweaks the cfs quota based on the number of cpus on the system and the requested amount.

—cpu sets the quota, there is is a —cpuset-cpu flag for cpuset and you can detect both by looking at the /sys/fs/cgroup

Hi I'm the blog author, thanks for the feedback

I'll try and clarify this. I think this is how the sympton presents but I should be clearer.

People using Kubernetes don't tune or change those settings, it's up to the app to behave properly.

False. Kubernetes cpu request sets the shares, cpu limit sets the cfs quota

You said to change docker flags. Anyway your post is irrelevant, the goal is to let know the runtime about how many posix threads should it use.

If you set request / limit to 1 core but you run on 64 cores node , then you runtime will see that which will bring performance down.

Original article is about docker. That’s the point of my comment - dont set cpu limit

I intended it to be applicable to all containerised environments. Docker is just easiest on my local machine.

I still believe it's best to set these variables regardless of cpu limits and/or cpu shares

All you did is kneecapped your app to have lower performance so it fits under your arbitrary limit. Hardly what most people describe as “best” - only useful in small percentage of usecases (like reselling compute)

I've seen significant performance gains from this in production.

Other people have encountered it too hence libraries like Automaxprocs existing and issues being open with Go for it.

Gains by what metric? Are you sure you didn't trade in better latency for worse overall throughput? Also, sure you didn't hit one of many CFS overaccounting bugs which we've seen a few? Have you compared performance without the limit at all?

Previously we had no limit. We observed gains in both latency and throughput by implementing Automaxprocs and decided to roll it out widely.

This aligns with what others have reported on the Go runtime issue open for this.

"When go.uber.org/automaxprocs rolled out at Uber, the effect on containerized Go services was universally positive. At least at the time, CFS imposed such heavy penalties on Go binaries exceeding their CPU allotment that properly tuning GOMAXPROCS was a significant latency and throughput improvement."

https://github.com/golang/go/issues/33803#issuecomment-14308...

Reply on news.ycombinator.com