Hacker News Reader

Comment by exe34 | Hacker News Reader

> What we need is a better framework for punishing bad software

So like engineering licensing and insurance?

I'm pretty ignorant on the ins and outs and trade offs associated with accreditation but yes that sounds like one solution.

It doesn't strike me as unreasonable, software engineers are now able to cause billions in damages and/or cause loss of life via primary or secondary effects.

Sure, one of software's greatest strengths was that anyone could learn it and it'll be sad to lose that but that's not really a reason for structurally enabling things like the Post Office scandal.

You probably don't need accreditation to serve cat pics but things dealing with money and life/death should. But IANAL so hopefully someone else can bring better insight to this area.

> it'll be sad to lose that

People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.

I think we need to start thinking of an individual's persona which includes all the information about them that is online or in government (non secret) files.

That needs to be considered when designing regarding regulations for software that interacts with someone's persona.

So anyone can set up the equivalent of an art stand in the park (serving cat pics), but if they start either selling cat pics or selling advertising that relies on collecting information about the personas on the site, then that needs to be regulated.

> People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.

You can even have non-licensed people doing work, and a licensed individual signing off on the end result. They would need to review the designs and work to make sure they agree with how the work was done, because it's their license on the line if something goes wrong because of bad craftsmanship.

It doesn't need to be too dramatic. It's already a general rule never to write your own cryptographic or financial code from scratch, you reach out for established libraries or services written by experts.

The law may require to use the ones that are certified if you are doing something sensitive. It doesn't need to be universal necessarily, but it should apply to public tenders for critical infrastructure at least.

You want some government mandating your dependencies...?

Yes. Legally mandated supply requirements have been an element in every other engineering sector for decades or centuries. If you're building an airplane you can't purchase from any random aluminum supplier, there's an authorized list.

That would seem to result in industry consolidation and declining innovation.

Software isn't made of physical materials anyway, it's speech. How about instead we impose open source requirements to enable public verification of critical systems?

Innovation like calculating numbers incorrectly and getting people sent to jail? Or maybe just another hundred slow, rent-seeking React web apps that'll disappear when the series B money dries up, if we're lucky. We must have different ideas of what innovation is.

So, outsource the core government function of regulation to "the public". Who exactly do you imagine is going to actually do this public verification, and where is their paycheck going to come from.

Some software is critical infrastructure and needs to be treated as such. We are not special. Every other engineering discipline has gone through this same process as and arrived at the inevitable conclusion that government regulation is essential, but only after causing unthinkable damage to the public first

software "engineers" love the cachet but hate the idea that they might also have to uphold the values and responsibilities of engineering.

I say we regulate the word engineer the same way it is in many countries for real engineering. if you don't want to progress beyond code monkey, you can be a software developer and innovate yet another react clone. if you want to be called an engineer, you learn and follow the regulations.

Forcing transparency isn't outsourcing anything.

Enabling public verification is not the same as required verification.

At least when you are building key infrastructure for the government, directly or indirectly through third parties. I don’t think it’s unreasonable.

And it is not as absolute as you make it sound. Only dependencies for specific critical functions may be regulated. And they don’t have to literally force a whitelist of dependencies on you, just whichever has been certified as appropriate for that purpose.

This is how you get FIPS 140 [1], which for those not in the know is a US Federal standard that mandates encryption which is _less_ secure the the current state of the art and has been for decades. (Yes, there's a new version which was approved 5 years ago and which is still rolling out [2]).

[1]: https://en.wikipedia.org/wiki/FIPS_140

[2]: https://csrc.nist.gov/Projects/fips-140-3-transition-effort

At the same time, turning on FIPs mode is the way we discover that some of our modules were using MD5 in security critical places. Because the government actually enforces FIPS, people (primarily Red Hat I think), now actually put in the bare minimum of engineering effort so that when you set fips=1, the system will actually enforce the policy (unless you go out of your way to override it, or use a non distribution provided crypto stack).

Sure, now that the infrastructure for this has been built, it can be configured to require stronger crypto then FIPS does, but that infrastructure would never have been built without the likes of FIPS, and the government mandating it's use. And I know this because even with all of the hard engineering work done of building that infrastructure, there are no commonly used stronger policies; because the only people who actually care are the ones forced to care by the likes of FIPS.

Our electrical standards might not the safest way of wiring buildings, and not what we would come up with if we wrote the standards today. But they are orders of magnitude safer then what electricians would be doing without the standards.

Who maintains the mandated dependencies? Who performs certification?

What prevents regulatory capture?

how does it work for aircrafts? how does it work for anti-money laundering policies?

The way it works for aircraft is that now it's getting prohibitively expensive to design new planes. But new planes are still needed, so Boeing found a solution, and made the 737-MAX which is in theory an upgrade that does not require re-certification, but is in fact different enough that the differences lead to hundreds of deaths.

Perhaps without the certifications lots more people would have died. I'm just an armchair analyst. Just food for thought.

Uh, the MAX was a disaster because Boeing is run by MBAs instead of engineers. Regulation is why air travel is remarkably safe. Those rules are written in blood, something everyone here conveniently seems to forget.

So you write regulations for software development, but software companies run by MBAs instead of engineers can still cause disasters. I guess as long as disasters are rare, it's a win.

Is Fujitsu run by engineers?

lots of people did die without the regulations!

Famously, civil engineering has never solved this problem, either, leading to the sad state of affairs in which we find ourselves today, wherein no bridges have been built anywhere.

I don't understand the confusion, practically everything we interact with in our daily lives is regulated and goes through certification processes, just look at how all that is done.

Food, drugs, healthcare, consumer products, chemicals, cars, planes, trains, buildings, utilities, energy, infrastructure, salaries, loans, investments, accounting... Even media requires some licenses, receives age ratings, and has restrictions on advertising.

It's not rocket science, this is normal for every single other industry.

And that has resulted in massive industrial consolidation and lack of innovation in food, drugs, healthcare, consumer products, chemicals, automotive/aerospace/locomotive manufacturing, construction, utilities, energy, payments, finance, accounting, and media outside "tech," where we manage our own dependencies.

Rocket science is one of the few industries that's actually seeing active innovation.

This isn't (or doesn't have to be) about laws specifying how all software has to be made.

What they need to specify is the standards for software that certain types of organizations can use. Like government agencies, government contractors, medical organizations, construction and engineering firms, and probably some other kinds of large private businesses, depending on their industry.

Basically, if the software your organization uses can cause the level of destruction that Horizon did, it needs to have specific certifications, or you can't use it.

In order for such software to be certified, it needs to meet certain clearly-defined standards of quality, potentially including having all the technical leads of some level (or just all the developers, depending on various factors) be licensed, and have their licenses on the line of something like this scandal occurs.

It's not a panacea, and it would definitely be an absolute bear to get the terms of all of it defined both clearly and in a way that is likely to actually produce a quality product, but IMO it is likely to be worth it in the long haul.

think of it as accreditation profiles. the highest is for life threatening cases like flight/medical, the lowest might be government expenses.

While I personally love that idea, I suspect it's too complex to fly in actual attempts to get something like this in place.

the same could be said for aerospatial regulations, and we still manage to do them. civil constructions are mindbogglingly complicated and we still manage.

all it takes is enough people to die, and/or for rich people to lose enough money and it'll become the rule.

Transparency and competition seem to work better than certifications and credentials.

Just mandate open source if using public money.

How's it working for those postal workers who committed suicide?

That software wasn't open source, was it? I don't understand how that's supposed to be relevant.

If someone prefers solution B to solution A, bringing up a situation that had neither is not a counterargument.

I think it needs to be on the individual-liability level. Blessing implementations from corporations is an environment for monopolies to grow and corruption to set in.

The Post Office scandal at its core was a political/legal problem, not an engineering one.

Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.

There is no expertise for software that you could delegate to aside from an entity using some form of integration test.

> Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.

This is becoming more and more important as the AI industry is pushing non-deterministic computing further and further. There will be court cases where the full automated decision chain will be called into question, and a default assumption of "well, we don't have any logs so we should assume the software did the right thing" terrifies me.

The magic word that always pops up to me is "liability."

Even speaking from an IC/non-managerial place, far too many technical problems in businesses are really just symptom-alleviation (or worse, performative look-I'm-doing-something theater) for a deeper problem which involves misaligned incentives for humans.

Some of the sites we work on are insurance based. We save every step of the calculation (inputs and outputs, e.g. rates and sub-totals). An administrative user can see the entire calculation from start to end, including overrides at various steps (e.g. a manual discount), and breakdowns per state or item insured if appropriate. This seems like the acceptable bare minimum to me, rather than just showing a magic number. And it definitely helps to expose bugs.

Sure, it doesn't need to up-end the whole field, but it is rather obvious that critical infrastructure should require significant compliance, particularly if it's being purchased in a public tender process where the cheapest supplier that fits the criteria is hired. Certifications like SoC 2 are already a big thing.

> > What we need is a better framework for punishing bad software

> So like engineering licensing and insurance?

How is engineering licensing and insurance punishing bad software ?

Microsoft is still going strong and all they do is "checklist security".

Reply on news.ycombinator.com