Hacker News Reader

Comment by red_admiral | Hacker News Reader

I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.

There is precedent for major power outages, a huge majority of which are not malicious: https://en.wikipedia.org/wiki/List_of_major_power_outages

I remember the day when the Swiss railway power network went down for a day (in 2005) because one power line was down for maintenance and someone pressed the wrong button and produced a short circuit somewhere else. It's a bit like the incidents in planes were one engine has a problem and the crew shut down the other one by mistake.

View on HN

I seem to recall a tree falling in Ohio knocked out the power to NYC.

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

A monkey took out the whole grid in Sri Lanka this year

https://edition.cnn.com/2025/02/13/asia/sri-lanka-power-outa...

> The animal had come into contact with the transformer at the station, disrupting supply to the entire country. There were no immediate details on whether the monkey survived the incident.

I'm sure the constituent atoms of the monkey survived.

I saw a video the other day of some human running and jumping on a transformer after hopping a fence, dancing on the transformer in a distribution site.

It ended as you'd expect, a bright light, a lot of curse words from the camera operator who was probably blinded temporarily.

Electricity does not care.

> Electricity does not care

Nor does Darwin

That's a horrible story.

I can't believe and I'm horrified someone actually published such a video.

I'm sorry, and i agree.

I still lament the passing of cybersquirrel1

https://cybersquirrel1.com/

If that's all it took to turn Srilanka off.Even if that country lags behind western Europe, what would it take for a few humans who intentionally cut off power.

Trying to stay on the facts, this incident is likely accidental but some people even the very workers at energy companies could send a message for, I don't know. A pay raise?

Isn't it amazing? In movies, evil terrorists take out the grid all the time (or threaten to), to terrible consequences, and they need genius hackers to do so. In reality, it's easy peasy and still, people hardly ever do it (and nobody (ish) dies if they do). Humans are good!

https://en.wikipedia.org/wiki/Winter_of_Discontent

Yes, you can try to hold the country hostage for your salary by going on strike, but that's the sort of thing that results in very energetic union-busting.

Actually sabotaging the infrastructure would result in terrorism charges, or at the very least the JSO treatment.

> what would it take for a few humans who intentionally cut off power.

https://en.wikipedia.org/wiki/Moore_County_substation_attack

Unsolved to this day.

I mean i'm pretty sure there are laws in place everywhere to avoid a single pay raise strike doing nation wide catastrophic events. Hospitals for example are usually limited in how they can strike.

That's assuming the people who want to strike care about having a job afterward. I know the state hospitals stop loss people when there's a disaster, so I keep abreast of predictable disasters and tell my wife and friends "hey you're sick, go get a doctor's note for the rest of the week" and then they're not stuck at the hospitals for days during a disaster.

Not that type of hospital.

What kind of person fakes illness expressly so they don't have to help out during an actual disaster when their skills are needed? "Whenever I hear there's a fire, I tell my firefighter friends to take the battery out of their pager so they won't have to go fight the fire." I agree we owe these people better treatment than just mandating they work heroic hours, but this only makes things worse on those who comply with the mandate.

My neighborhood and community is more important than the hospital; the friends i mentioned specifically are also radio operators, so they have more important things to do during a disaster than be a warm body at not that type of hospital. A sanitarium.

The low-wage workers can get triple-time and take weeks off "later in the year" for covering the stop-loss, so they also benefit.

I don't call fire fighters, because they are emergency responders. Please note i said what i do, not what i think other people should do. The people i tell this to are free to do with that information what they want, i don't demand it or anything.

Consider this a contextual error on my part. It's more of an "inside baseball" snippet of conversation than anything that requires judgement.

Thanks for explaining what I didn't have the context for.

That wasn't tree falling in Ohio, that was overloaded line sagged and shorted into a tree, compounded with several other factors that contributed to the grid instability and the inability of the grid operator to realize how unstable the grid was.

"Same difference" to use an Irish turn of phrase. If a short to ground takes out power to an extensive area does it mater what touched what?

Yes, it absolutely does.

A tree falling can be addressed by vegetation management and trimming. A power line sagging because of excess heat is operator error.

These are not remotely the same.

The proximal cause was a race condition in a piece of software, so as usual it comes back to you keyboard ticklers!

> keyboard ticklers

I hate this term, and look forward to using it all the time.

I think the point is that neither of those events should take out a whole city; the design is such that there is considerable redundancy in the system.

Indeed. For instance, power grids ideally operate with N-1/N+1 redundancy, i.e., the disablement of any single power line should not cause a cascading failure.

Relying on vegetation management along millions(?) of miles of lines is the definition of thinking you're in control without even knowing what risks exist. I see this every year in Texas, they trim back trees thinking they have solved the problem - except - they are allowed to trim them back I think 6-8 feet from the lines, ignoring the fact the trees are 20-40 feet taller than the lines. Nobody seems to see the risk that they left there but it's quite obvious to me, we're not eliminating risks at all. We're minimizing them at best, and even then the labor required to trim all the branches around lines every year is impossible to keep up with from an economic feasibility standpoint.

They trim back what poses the highest risk with the budget they have.

As you have identified A wider right of way costs more.

Usually for lines above some voltage, perhaps 200kV, the cost of an outage due to a tree strike outweighs the cost of additional vegetation management so they will clear the right of way wide enough that no tree can fall and hit the power line.

Around here for 130kV the right of way is still as narrow as it can be and we annually take down the riskiest trees as this is the best for our budget, which is not unlimited.

Concur, they have huge fire breaks they put high tension lines in, where the easement is easily 100' to either side of the lines. This is in Louisiana. When there's a hurricane these lines still break, but not because of trees. During heavy storms, smaller trees hit smaller (480? Some KV?) Lines that go along highways to residences. A high tension line down means a few days without power (about 3-4), a tree on a lower voltage line is usually fixed within hours.

They go through and remove damaged trees near the easements of the highway lines, as well as branches that could break into lines.

As an aside we lived on the same section of grid as the sheriff, and our power was rock solid for a few years, then he left office and now our power is better than average (at least better than our neighbors who's power line cones from the other direction).

> During heavy storms, smaller trees hit smaller (480? Some KV?)

Residential distribution voltage varies by utility but it’s usually in the medium voltage range, 5kV to 35kV, with 13.8kV being common.

Where I live, there are no trees in the path of high-voltage (400KV) transmission lines. Everything is cleared below them and the voltage lines are about 90-120ft high so even if some trees grow, they will not touch the lines.

It seems like both are a monitoring problem and both have a maintenance fix. A tree falling is more forgivable though.

Both of these are operator error issues.

Is “same difference” more common in Ireland than in Britain? It’s certainly common in the US and seems to have originated there: https://english.stackexchange.com/questions/277142/etymology...

Definitely not an Irish turn of phrase. An unfortunate import.

The tree had malicious intent. You can't convince me otherwise.

Can you blame them? Just look at how they've been treated in the last few hundred years.

I vaguely remember an episode of some thriller series when I was too young to be watching it, perhaps Twilight Zone (or similar) where someone was hearing screaming, they went out a d saw trees being cut.

They've had a grudge against humanity since the fall. Ever since—once a year—they send a clear message, "leaf" but do the humans listen? No.

There was also a race condition in some management software that hadn’t tripped in over thirty years of usage.

Having lived through the Texas electricity fiasco in 2021, I would blame cost cutting, the reckless drive for “efficiency”, and maximizing shareholder value.

In Texas, the electric providers cut staff and maintenance to maximize shareholder value. They will not have redundant systems and redundant plants out of the goodness of their hearts. The Texas marketplace actually allowed them in the odd event of an outage to charge astronomical spot prices thinking this will incentivize them to have redundant systems. This was a foolish fantasy.

Now in Texas, discussion of how to cost share redundancy have taken place. But no one wants to pay for it. https://www.texastribune.org/2023/03/01/texas-power-market-p...

Show me the incentive and I'll show you the outcome.

If you want a no fail grid you need to incentivise a no fail grid.

> were one engine has a problem and the crew shut down the other one by mistake.

So frequent it even has its own wikipedia page: https://en.wikipedia.org/wiki/Category:Airliner_accidents_an...

“induced atmospheric vibration” on 400KV lines is the current theory

Probably didn't help that before the outage hit, Spain was running its grid with very little dispatchable spinning generation, and therefore not much inertia.

Solar PV/thermal + wind: ~78%

Nuclear: 11.5%

Co-generation: 5%

Gas-fired: ~3% (less than 1GW)

> Solar PV/thermal + wind: ~78%

This is (a) incredibly impressive to achieve and (b) definitely the point at which the battery infrastructure needs to catch up in order to reduce the risk of such incidents.

> battery infrastructure needs to catch up

This non-existent technology will surely catch up very soon, I wonder what takes them so long.

The only battery available at this scale is hydro and it doesn't do very well in Spain because of droughts.

It definitely does exist, I don't know why people are weird about this: https://www.solarpowerportal.co.uk/zenobe-200mw-bess-operati...

I was just thinking that grid scale batteries could be extremely useful in a black start situation.

> very little dispatchable spinning generation

Makes the case for favouring flywheels over batteries.

Just to clarify this a little bit… flywheels are cool for absorbing transients and providing short-term hold-over during a blip but don’t have great long-term capacity.

http://claverton-energy.com/active-power-article-flywheel-en... as a smaller scale example. $330/kW but at that price it only has 15 seconds of carry-over, basically just enough to get the diesel generator fired up.

Spinning reserve in the grid is equipment that capable of long-term generation very quickly. In the case of hydroelectric dams, they will often cut off the water supply to some of the turbines and use air pressure to push the water out of the way; the generator attached to the turning essentially turns into a motor and keeps the turbine spinning. If you need to bring it online, you open the water valve and let the air out.

Similar situation with natural gas-fired simple cycle turbines. They’re sitting there running at low output. Need more? Just add fuel. For combined cycle it might take a bit for the boiler to warm up for full output but having the first stage running full tilt will get it warmed up fast.

Yeah, but the angular momentum of turbines/flywheels gives you a few seconds to cutoff consumers before the grid collapses.

In a solar grid you probably have milliseconds instead of seconds, this could be the reason why the automation failed in this case.

That's part of the UK's plan:

https://www.proactiveinvestors.co.uk/companies/news/1057463/...

Same all over, apparently we had the biggest in the world a few years back, I’d imagine it’s been overtaken by now https://www.siemens-energy.com/global/en/home/stories/irelan...

Together, the flywheel and the synchronous condenser have an inertia of 4,000 megawatt-second

That's insane, imagine if it let go.

In electrical terms 4 GJ is 1,111 kWh. That's about 15 EV batteries' worth, or about £220 worth of retail electricity (at 20p/kWh). So it's a lot compared to the usual domestic things people are used to, but not much in grid terms.

Or if you consider the Irish grid (average consumption around 5 GW) that's enough energy to power the grid for about 0.8 seconds (obviously it's not going to have enough instantaneous power output to do that, but again for a sense of scale).

If Ireland had 10 of them, that'd be 8 grid-seconds worth of energy. Although, of course, actual disturbances aren't going to be that large. A few percent imbalance perhaps?

So if the whole grid had an instantaneous 10% imbalance, one of those units could carry it for 8 seconds.

(EDIT: changed energy numbers to fit the appropriate power grid)

flywheels don't really work synchronously, though. Or at least, if so, they're not very useful as storage. Inverters can simulate inertia just fine, in fact they can simulate a much larger inertia than the corresponding power of spinning generation, leading to greater grid stability (whether connected to batteries or solar/wind, though without batteries its a little 'brittle' if you run near 100% output, as the power needed may suddenly disappear, so this is probably not good practice for the system as a whole. And you still need some give in your inertia so the grid can actually communicate about supply and demand)

flywheels are batteries, of a different kind (and density) than lithium or lead ones.

You know what he meant. Respond to the charitable interpretation of the comment, etc.

They're usually almost more capacitators than they are batteries from an electrical standpoint. They're there to rapidly smooth out the smaller jitters.

I would _love_ to understand more about how this atmospheric state impacted the vibration of power lines. Sounds exotic.

I’m no expert but I think it has to do with this effect: https://en.wikipedia.org/wiki/Vortex-induced_vibration

Edit: might be a completely different kind of oscillation than I was thinking of. https://news.sky.com/story/spain-portugal-power-outage-lates...

So I would also like to understand how this works :)

Update: BBC news says something something hot weather, EU council says it doesn't look like a cyber attack.

As confirmed by António Costa (President of the European Council) on X: https://x.com/eucopresident/status/1916859055546544517

Hot weather? Huh? It's spring

Spring is nowadays the best time for the grid and renewables. Low consumption as heating period is coming to and end, air conditioning is not required yet and the sun can be strong already providing lost if solar power output. I can imagine with a grid and a huge growth of solar generation that there are a myriad tripping stones that can upset the balance.

For areas utilizing hydro, spring tends to bring the strongest annual flows due to snow melt.

In the UK we're having one of the hottest springs on record. Spain and Portugal may be similar.

Might as well mean that they suspect the culprit, but they're not going to do anything anyway for this specific culprit, so let's all pretend it was an accident. Why make yourself look like a fool?

They already became a laughing stock once for promising the "strongest possible response" for the Nord Stream 2 sabotage [1].

[1] https://www.reuters.com/business/energy/eu-sees-sabotage-nor...

>an incredibly complex system that can fall over even if no-one's being stupid

I only have a layman's understanding of power grids, but I thought they were incredibly hardened, with backups and contingencies in depth

Are the grids at this scale really this brittle? Would there be a death toll from this?

I also wouldn't blame malice without corroborating evidence

> but I thought they were incredibly hardened, with backups and contingencies in depth

Some are harder than others, and some have random flaws which nobody can really predict.

Spain seems in the transition to renewables, so it's possible that they have some flaws because they are still in the process, or because it's something which never happened before and is unknown territory. Also, Spain had some economic problems in the last decade, maybe someone build to cheap or was even cheating somewhere.

> Are the grids at this scale really this brittle? Would there be a death toll from this?

Hospitals should have backup-systems. Traffic should be able to stop in time. I guess the most problematic parts are people stuck in elevators and other spaces which only open electrical, as also the loss of cellular phone-connections for calling helpers.

> the loss of cellular phone-connections for calling helpers.

All the mobile phone installations that I saw had power for at least 24-72hrs depending on how far from civilization they were. The carriers have backups and everything.

The problem in these kind of situations is the saturation of the mobile network, not its availability.

Hardening focuses first on not damaging equipment and second on providing energy. If things go wrong quickly enough you don't have time to react, because after a power plant disconnects you get sudden bumps in load that can trigger a chain failure near the original point. The last time it happened in Europe was in 2003, which isn't too shoddy.

What Spain's PM is saying (and is being reported by Spanish newspapers), 15 gigawatts of energy production went down all in 5 seconds. Hardening to tolerate that much of a change, that fast is a more extreme event than a grid the size of Spain's preps for.

To be fair, a big deal of those 15GW probably went down as a result of the initial outage: equipment is designed to intentionally disconnect completely when the grid is behaving badly, rather than trying to force it into submission and potentially causing serious equipment damage.

In reality this means you might lose, say, 1GW due to a transmission line failing, have a big frequency dip as a result, and then have 14GW drop offline like dominoes because they sense a grid frequency outside of safe operating parameters; disconnect as they go into safety mode; cause the frequency dip to worsen; and pull even more plants offline with them. If you're not careful, a small outage can quickly cascade into an entire grid going offline.

Not many month ago the Italians trains was disrupted by a single nail into a cable: https://www.reuters.com/world/europe/italian-rail-services-d...

> I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.

I think we should prepare for the worst though. It's wrong to assume it's not an attack too, and until we can conclude it's not an attack we should be prepared to deal with the possible consequences and act accordingly.

I think James Burke's classic talking about the fragility of our complex interdependent systems starting the episode from the 1965 Northeast blackout is still relevant and an interesting watch: https://www.youtube.com/watch?v=XetplHcM7aQ

Every single retrospective I've seen on major power outages is full of miscommunication and people who would be fired if they worked in fast food. As well as blaming the tech boogeyman ("Our systems were on the fritz this morning")

There’s an explanation that earth’s magnetic field is now so weak that a normal solar wind ended up breaking through a vulnerable spot.

Very poorly explained right now by Space Weather News. I am waiting for an updated explanation.

Seems to be caused by fire in France that connected Spain / France

a.k.a. Hanlon's Razor https://en.wikipedia.org/wiki/Hanlon%27s_razor

As far as we know stupidity isn't a primary cause, and local authorities are also probably competent at scale.

On the other hand this is Spain we're talking about, the expectations of people giving a fuck about anything are slightly lower than average.

Or that time in 2003 when a tree fell on a power line in Switzerland and all of Italy ended up without electricity.

We live in a society filled with systems that can have "normal accidents".

I'm going with AGI awakening in some lab today.

A few months ago I was hit by a blackout literally the second I was about to start delivering a company-wide talk on AI. Everything went out - Internet, mobile networks, street lighting, the lot.

We're a remote business so it seemed like I'd just rudely dropped off the call, but as everything was down I couldn't let people know what'd happened.

Apparently it was caused by botched maintenance work affecting 30,000 houses, but the timing was so perfect I can't help thinking it was because our AGI overlords really didn't want me to deliver that talk for some reason.

Sorry I couldn't resist: https://www.youtube.com/watch?v=jnDk8BcqoR0

But there is so much highly motivated malice right now.

Reply on news.ycombinator.com