Hacker News Reader

Comment by andrelaszlo | Hacker News Reader

Funny, similar patterns are super common in API traffic. A large spike at midnight UTC, a smaller one every hour, even smaller ones every 15 minutes, and a little bump every minute if your metrics have enough resolution. Of course, if someone calls your API every minute you should probably consider some kind of streaming, or at least include make the request cheap by requiring a cursor of some kind.

View on HN

Package managers like apt will delay automatic updates for a random time to avoid this problem:

    RandomizedDelaySec=12hr

https://wiki.debian.org/PeriodicUpdates#Modify_the_schedule_...

Systemd timers also have randomization features. They're very useful.

Opscode built in this feature into Chef when they first released it, oh so many years ago. They learned their lesson the hard way.

Yeah, adding 'jitter' for periodic tasks that make remote requests was standard procedure when I worked for a large CDN. Every cron entry on every server would run with a delay of between 0 and 59 seconds, based on a hash of the server name. You basically 60x your internal infrastructure capacity by doing this (depending on how long the request takes to serve, of course, as well as a few other factors)

That, exponential backoffs, and local cache for external resources needed to operate the CDN (such as configuration) were the foundation of running a large network reliably.

> and a little bump every minute if your metrics have enough resolution

Amusingly, you can also observe periodic bumps if your metrics don’t have enough resolution, due to aliasing effects dominating signal (guestures vaguely at Nyquist frequency).

By definition, if you can see effects then the metrics have enough resolution.

The specific resolution is the Nyquist frequency - that part is accurate.

Sampling metrics is sampling and leads to aliasing.

Similarly, sample count/interval limits in tsdbs like prometheus are not downsampling. They're just skipping samples. This also creates aliasing. Can be very noticeable especially if you combine it with rate/derivative functions.

I've just convinced my team to be nice and add a random delay for all those periodic checks in our software last week :)

Yes of course, there was the check at 2 am local time before ...

Just adding random shift of the duration of the check already will get you uniform distribution

Customer wants every 12 hours; customer wants uniform load on the server. You do a refresh at startup, one after a random (0-12) interval, then every 12 hours. Is that a random shift?

Same here. We see this kind of spikes in our tunnel traffic also - every hour and specially at midnight UTC. Cron jobs everywhere :)

Jenkins server has the feature to randomize job start time since at least 15 years. One would've thought this is a well known problem.

I wished cron had a similar feature. I missed that on acouple of occasions.

Even more important if your devices are dialing into a modem bank. We spent a lot of time designing retry schedules and such when I worked on these (ironically for energy storage/demand management devices)

Reply on news.ycombinator.com