Hacker News Reader

Comment by dylan604 | Hacker News Reader

Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.

View on HN

The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.

Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).

I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.

That still lands it at the top of the minute, which is AFAIK the pattern the researchers found.

Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:

  * * * * * sleep 17 && …

etc.

That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.

Systemd timers do this natively btw, no need to hardcode And imho are also friendlier.

Timers also let you logical-AND elements within one simple declaration. Combine that with a random delay, and you can easily define a trigger such as "the 2nd Saturday of the month, anytime between midnight and 6AM":

  OnCalendar=Sat *-*-08..14 00:00:00
  RandomizedDelaySec=360min

You can tell a timer whether to re-use that same randomly-selected time during subsequent runs (FixedRandomDelay), or to re-roll the time for every trigger. They can also persist their last-run history across reboots, and trigger if they missed their previous time-slot.

The improved logic vs. cron is one of the features which makes my life easier, and so I started using systemd more often. IMHO some of the newer additions aren't quite as solid; but the basic units have proven to be very handy. Another example: The 'path' unit, which can trigger whenever a defined file is updated.

https://wiki.archlinux.org/title/Systemd/Timers

BSD cron has supported jitter for decades. This is strictly a linux problem.

It's a human problem. Noone "expects" their own "minimal" checks to compound. And everyone (re)implements periodic checks in ever-more-inventive ways. BSD cron solves this ? Alright. Does prometheus use it to schedule its periodic polling ? Do OTel collectors use it to smear their backend metrics uploads ? Does your own code use cron jobs for its "periodic" heartbeating ?

If the answer to that is all-yes, then my compliments to you. Alas ... at least my experience says that's not how things are done.

But systemd bad :[

People slowly are getting used to it. (Less so with journalctl)

More like people were forced to use systemd and gave up due to lack of choice, along with an entire generation not old enough to know SysVinit.

The "forced to use systemd" meme is a weird inversion. No one complained about being "forced" to use sysvinit when their was no choice, but when distros started to use systemd, which still allows to use sysvinit style services, that's suddenly framed as an infringement of free choice.

journalctl is good idea executed terribly. It's builtin data format is massively wasteful and not even good at searching for data you want (often need to go thru all the files anyway)

Journalctl with SQLite or a columnar embedded database would be amazing for sysadmins. Current format is entirely useless

That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?

Our configuration management system(Puppet) has function for that, fqdn_rand

give it number and you will get random value up to that number, same every run for a given host, but different per host (basically, hashed host fqdn)

add a label and you will get different number but similarly consistent. Makes it real easy to randomize run across the fleet.

Would be easy script to make.

Also systemd timers have the randomization builtin, just need to specify time

    RandomizedDelaySec=12h

Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.

Reminds me of hunting tooth gearsets. Perhaps these systems should be checking battery every 59, or 61 seconds, as those numbers are coprime with 60.

https://shoppress.dormanproducts.com/hunting-tooth-gearsets/

By default, systemd timers activate at a random interval within 60 seconds of when the timer is actually set in order to solve that potential issue. It's weird when you set a 60s timer and it activates twice within a few seconds.

Yeah, and it's not cron, it's "everything running on a digital schedule"

Analog systems naturally have a dispersion to them, digital ones, especially with everybody on NTP, much less so

Reply on news.ycombinator.com