Hacker News Reader

I get a disturbing amount of equipment that does not take into account the inrush current of starting many loads at the same time. While each individual device is protected by a breaker that can handle the inrush of it's load the main breaker for the equipment is not sized to handle every load starting at the same time. The fix is to include logic that starts things up one at a time but convincing engineers, project managers, and machine owners that they didn't think of everything is an uphill battle.

what? Inrush current is well known in electronics engineering.

In other words, convincing ignorants is an uphill battle. Those who know what "inrush current" means don't require convincing about the value of rolling starts.

I assume you are talking beyond what (in the DIN rail world) type B/C/D breakers are meant to handle?

It's sort of a solved problem - the ones I'm aware of are things like LynTec sequenced panels built exactly for this, and Siemens' SENTRON semiconductor CBs where you can control then curve to your spec.

Convincing stakeholders though, is almost a job of its own.

These were all across-the-line, 3-phase 600v starters so there was no option to slowly start things up. Soft-starters would have been ideal but they are 5x the cost.

We used to have a turbopump that would trip the breakers every time you turned it on from cold. The pump seemingly attempted to spin up to max RPM before settling down to a more sustainable load. The only way to get the thing to stay on was to turn it on once, let it warm itself up on the initial attempt, trip the breakers, and try it another cycle now that it had some momentum and would pull slightly fewer amps. Sometimes took a few attempts to get that rolling start working.

That's a great way to lose the breaker though. If you're lucky the latch will fail. If you're unlucky it will weld shut.

Not ideal, but it was the only thing that worked. This was attached to a million dollar piece of equipment, so any remedial electrical replacement work would have been slightly inconsequential. Thankfully, it was one of those machines that you turned off as infrequently as possible, so we did not have to do this procedure too many times.

The vendor did get the hint, and the second iteration of the device did slowly warm up the pumps so this process would no longer be required.

Yes, soft start is the way to go. Also: proper industrial breakers allow for temporary overload. I have a couple of pieces of gear that have inrush currents well over the limit of a normal breaker so I put them on a slow one and now it is fine.

https://onccy.com/how-to-choose-between-type-a-b-c-and-d-min...

> Also: proper industrial breakers allow for temporary overload

Notoriously a crescent wrench makes an excellent 1000A slow blow fuse

A 5/32" flat-blade screwdriver makes an excellent fast-blow fuse when dropped across exposed bus bars. And you can re-use the remains of the handle to stir your coffee once you've retrieved it from across the room.

Soft-start is in theory the way to go, but it's very dependent on the motor type, there's a whole pile of different methods required for induction motors depending on whether they're capacitor-start, permanent split capacitor, shaded-pole, etc, so you can do it in some cases by lowering the voltage, or with a VFD, or a VSD, or the old-fashioned way with a centrifugal switch but that requires very careful handling. If as the OP said it's running million-dollar equipment you probably don't want to monkey with it at that level.

Reminds me of computer fans, some bioses seem to do that ramp up for no reason thing as well.

Fans have a minimum speed needed to spin up when cold starting from 0 RPM. So it's safer to start at near max RPM and spin down to the desired target RPM if the controller doesn't know the exact operating characteristics of each fan.

You can observe this with a speed control knob/ESP32 where a running fan will happily run forever with the knob turned down to e.g. a 30% duty cycle. But after turning to 0% and waiting for the blades to fully stop, if set back to the same 30% they won't move at all or you'll maybe hear a sketchy clicking/whine sound with the blades intermittently twitching.

I've always assumed it was to calibrate the PWM sensor for fan speed, so it starts with no sensing active, runs up to max, and then the sensor kicks in and settles it down. You don't need to run a fan at max speed to get it to start, most will start with very little power applied.

not exactly relevant to computer fans, but in case of your usual standing fan you need a way larger kick to get it going. if it fails to start but stays powered it will stay stuck, heat up and fail. this is why fans usually have the highest option as the first one, so you have to start it at full speed at first, and only then reduce it.

You also want to be sure that, in the absense of fan-control, the CPU won't overheat. Full-on is safer than nothing until a driver is loaded.

Absolutely fascinating comment regarding gears and coprimes. This is the first time I have heard of either.

My mind goes somewhere completely unrelated to the comment though: cicadas!

I believe it is a mystery why some species of cicada have have a 17-year life-cycle. Theories have included avoiding predation etc.

However, I wonder if the 17-year cycle (prime) avoids "even wear patterns" somehow in their DNA??

coincidentally, earlier on HN: https://news.ycombinator.com/item?id=50333323

59 seconds will still have second alignment. Maybe it needs to be prime milliseconds or microseconds.

It depends on a lot of factors, but even second granularity gives you a 60x reduction in the thundering herd. Whether more is needed or even possible often depends on the specific problem domain. For example, if your task takes 20ms, us granularity doesn’t help. Indeed it makes it worse if you’re using randomization because there’s a lot more microseconds than seconds so you can end up with a lot of work that ends up in practice still getting scheduled together due to other delays.

You want your jitter to be significantly >> than the time your task takes as otherwise you get non intuitive degradation that can reintroduce the problem you’re trying to solve. It’s a balancing act picking the jitter that drops off the thundering herd enough and one that is still greater than the time your work takes.

But seconds are 60X higher frequency than minutes. This is almost 6 octaves higher. The grid will low-pass-filter that out much better than the minute period. (You can see that trend in the diagrams given in the article: the amplitudes all get lower with decreasing period to the left, where the 60 second blip is a an outlier.)

Secondly, the amplitude drops. Events that were concentrated on the xx:xx:00 times are now distributed into xx:xx:xx times. If the overall number of events is the same, each :xx time slot now receives 1/60th of the events that were concentrated on the :00 time. If power decibels are relevant here (20 log 10 x), that's a 35 dB reduction in amplitude of the driving signal.

35 dB less amplitude, shifted to a 6 octave higher frequency that the grid doesn't react to very well can be expected to basically make the problem vanish.

This is much like the situation of soldiers avoiding rythmic marching when crossing a bridge to randomize their footstrikes.

I think you should re-read the post you're responding to. It said 59.XXX seconds, and given the reasoning immediately after that, it's clear that XXX is intended to be a randomly chosen number of microseconds.

That was edited in later

Small nit: Wait(50+rand(10)) is probably closer to what you want to get about a minute between executions. As stated your proposed execution has an average about 30s execution interval with a variance of about 17s. If you’re talking about a random sleep on a 1 minute interval, this actually results in a possibility of 120s between tasks, also not what you want.

But yeah, as soon as you start using Rand most people make all sorts of mistakes. And if you’re using C Rand, don’t. It has other statistical problems.

He’s still allowing for exactly 60s between executions. He’s saying stagger the startup by a random amount up to 60s. Then we get what we want every 60s, but they’re all just staggered a random amount.

My bad. I misread it. Delaying the start like that does divide the thundering herd peak by 60. The point about c Rand stands.

The other thing to watch out for is you’ve still got 1/60 of the thundering herd at the second level. If the tasks are <1s and you have enough of them (as you probably do at grid scale), then doing it finer grained is warranted like offsetting the timer by a millisecond sleep of 10 * uniform_distribution<int> between 0 and 6000. That then gives you 10ms granularity, distributing the workload by two more orders of magnitude.

I almost wrote 59.999 seconds too, but I figured someone would nitpick that the timing often won't be that accurate or will get coalesced. ;)

I should have said, you almost always want to use the randomly staggered timers on initialization, not just when the exact timer interval matters! It's also important to test electricity or backend outages, which can unintentionally synchronize all the timers.

You have good advice and I missed the “on startup” for the wait.

A common aggregation is about 10ms although most OSes can give you finer granularity accuracy if you ask for it and have permissions. It really depends on how long the task takes and whether there’s any non-uniformity in its processing to make finer granularity helpful or harmful.

Unfortunately some people do read advice like it’s mana from the gods instead of thinking through the implication on their specific problem.

Coprime meaning "no common factors".