Little's Law is fundamental to queueing theory, but there's also the less well-known Kingman's formula, which incorporates variability of arrival rate and task size [2].
[1] https://www.johndcook.com/blog/2009/01/30/server-utilization...
Little's Law is fundamental to queueing theory, but there's also the less well-known Kingman's formula, which incorporates variability of arrival rate and task size [2].
[1] https://www.johndcook.com/blog/2009/01/30/server-utilization...
0-60 and you're still at minimum latency. 60-80 you're at twice the latency but it's probably worth the cost savings of the extra compute density since it's still pretty low. Higher than 80 and things are already slowing down and getting exponentially worse by the request
The first one is even worse; by 80% you're already seeing twice the delay of 70%.
If I were to describe the second chart I'd say 80% is when you start to get into trouble, not just noticing a slowdown.
I said minimize latency, not optimize latency.