Hacker News Reader

Comment by jrockway | Hacker News Reader

I guess my takeaway from this is if you're a government building some sort of asset you're going to transfer to the private sector, overengineer it. Nobody is ever going to do maintenance, so your thing essentially has to last forever. Imagine that they made the hooks so they only lasted for 10 years -- we'd have 10 times as many billion-dollar fires. But some engineer in 1918 said "you know, let's make it a little bit bigger, it's only a few cents extra" and they probably saved some lives in the process.

(Yes, it failed, but you have to admit that it had a pretty good run! 97 years is not bad for a moving part!)

View on HN

Paradoxically though if it failed every 1 year then pretty soon maintenance would have to be performed every few months or days or else there would be so many failures it would be really hard to ignore.

This is an interesting argument. Maybe 10 years as a lifespan would be better than 100 years since it'd force checks every year. With the 100 year lifespan, people will always assume it can be done next year instead of this year.

Reminds me of Let's Encrypt issuing 90 day certs instead of 1+ year so that you're forced to find a better automated solution and not have a panic when your cert expires in 5 years

I wonder whether or not that's the global maximum. I still occasionally run into sites that never figured out how to use Let's Encrypt, so they don't encrypt at all.

I personally achieved completely automatic renewals (https://jrock.us/posts/rotating-envoy-certs/) but it involved some work. (It also works so well that I don't even believe it. I have to go manually check from time to time because my monitoring never shows me any errors, and there are so many moving parts that I am continually amazed that everything works perfectly.)

Other people just aren't willing to do that. They like their nginx/Apache sitting on a lonely Linux box that they ssh into once every 5 years. The cost of changing any of that software is high, and they don't have a good system for running new software, like Certbot. We definitely didn't fix that.

Chaos monkey to the rescue.

Then they would say "this is annoying, let's make it to last long" which brings us to now...

"over-engineering" something can't make it last forever, can't prevent it from eventually wearing out. If that's the plan, it just postpones the day when people suffer. As you say, these things were engineered for the long haul but the people harmed in Santa Rosa fire that the collapse of a pylon caused probably aren't impressed by shear lifetime.

Father: Son, some day, all of this will be yours. And since we don't expect to do maintenance, some other day, it will explode and kill you or your grandchildren.

So what's the right answer then?

If you build something to last 100 years, you leave little incentives for maintenance and all but guarantee a catastrophic failure a century from now.

If you build something to last 10 weeks, you're creating a lot of waste - wasted labor, money, energy and natural resources, involved in producing and shipping replacement parts, as well as performing the maintenance job. You're also making everything that depends on your construction more fragile - the system is always a couple weeks from catastrophic failure, so it won't survive a localized crisis - like a natural disaster, pandemic, economic problems, maintenance company going bankrupt, etc.

So what's the correct trade-off function[0] here? How can we minimize waste and maximize reliability, while ensuring necessary maintenance happens on time?

[0] - f: specifics of project -> trade-off

Are we talking about k8s?

I jest.

Regardless, it seems ‘waste’ is what is needed to drive the economy.

Wasted labor is still people employed, getting a paycheck, and participating in the economy.

I don’t know if we want to eliminate waste without UBI or some other social safety net.

If we were to eliminate waste, less people would have jobs.

> If we were to eliminate waste, less people would have jobs.

... or more people would be available to take on work that's more needed and not currently being done. Otherwise, why have machines and technology, if they all primarily automate tasks your fellow laborers could be doing?

I think that work which exist only to provide jobs is a waste at societal level. But it's also a waste at company level - why would you make your product labor-intensive to maintain, where you could make it robust instead? In reality, the trick is to ensure you're not the one paying for maintenance - and then you save money by making more fragile products.

If you build something to last 100 years, you leave little incentives for maintenance and all but guarantee a catastrophic failure a century from now.

-- My comment about a family above was sarcasm, not a picture of reality. A large organization is quite capable of determining what maintenance is needed for its infrastructure and doing that maintenance. PG&E knew quite a while back that they were in trouble here. It wasn't the long time frame but profit and regulatory incentive structure that drove their decisions. They'd have done the same for short-lived infrastructure. They've done the same thing in the case of tree-trimming; something they should be doing constantly and haven't (note, living in Portland, or, the local utility, Portland General Electric (called PGE for kicks), trimmed trees with abandon, which is better than the alternative).

Sure, designing for maintainability is a nice strategy but fixed lifetime + replace is an effective strategy too.

Basically, the whole line of reasoning about short versus long lifetimes is a complete red herring and the neglect and decay of PG&E's infrastructure was an explicit on the company's part and documented as such and has nothing to do with how infrastructure was designed.

>So what's the right answer then?

Fail safely. In this case the failure of the beam should have broken the cables or fallen in such a way to not create a pathway across the non-zero phases.

The government didn't build this asset and transfer it. Great Western Power Company built it, which PG&E bought from a New York holding company in 1930.

I feel like an easy solution to this could have been an additional safety wire. The safety wire will catch the insulator and thereby give maintenance crew a few days to arrive before the tower fails for good. There is also a general maintenance problem that could be solved by installing inspection cameras directly onto every tower. Those cameras may have to be replaced more frequently than the actual tower components but I'm sure they would not only save on inspection costs but also allow them to respond to falling infrastructure much faster.

Of course PG&E has a cultural problem, not a technical problem. So I doubt they would do this at all.

I also found it bizarre that your average go-kart track (not to mention aviation) has the presence of mind to safety wire everything but a multi billion dollar utility company didn't

A century ago, when it was designed, they've probably correctly assumed there's no way a safety line, even unloaded, will survive longer than a hook in there. The hook was a robust solution that could easily last a decade or two.

I doubt people back then were expecting this to be left unmaintained for 97 years.

So the government has to spend more, look more inefficient, and then when a private corporation and runs it dangerously at minimum cost, it plays right into the "private companies are more efficient, we should privatise more" lobby.

If they only lasted 10 years, it would have been replaced with something easier to service by now.

>" your thing essentially has to last forever"

Wtf is the role of the private sector then? What are they contributing?

Reply on news.ycombinator.com