Hacker News Reader

Comment by yodsanklai | Hacker News Reader

> We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people and we can iterate and patch things quickly

I think you have things backwards. Devs aren't clowns who make it this far. But in industries where buggy code doesn't kill people, devs are expected to ship fast and iterate, because they can and more importantly, it serves the business.

Personally, I'm all about good software engineering principles, unfortunately I don't think you can last long in most companies if you're too opinionated about this and can't compromise (and it's getting even worse with AI, reading the code is no longer a requirement, go explain your teammate that there's an abstraction leak in their code...).

View on HN

Yes... there are statisticians on my team, for example. They're capable, thoughtful individuals however have little exposure to the oblique and (outwardly) contradictory ways in which production software fails.

But explaining that openly, as is your duty, means surfacing something which could be read as 'bad work' by a non-technical manager unable to grasp the complexity. So in low-trust teams it very quickly becomes a game of brinkmanship, or a prisoner's dilemma for technical diligence

The problem is the "interface" of software engineering being oblique to outsiders. It's alchemy in essence.

The logical extrapolation of AI capabilities stress-testing human software is all bugs being discoverable. Hence, you have to aim for actually bug-free software.

The only way to do that is to make your code provable. That again is only realistically doable using AI.

In the process, people will have to turn the current mere pretense of software engineering into reality, making the engineering part explicit.

That in turn only works when humans actually understand what they're doing and can communicate it to outsiders.

I am currently in a stalemate where I have written a proposal that is very technical. The only “business” outcome is that we clean up a massive security mess and rebase on a new consistent architecture.

My manager can’t understand the RFC, and knows that clean up projects aren’t palatable to his senior eng leadership so he keeps asking me to simplify or derive some business justification. I’ve proposed superficial ones like improving security posture etc but he wants a hard number that can be measured (yet another senior leadership mandate).

I’m kind of exasperated and honestly think it’s his job to figure this out.

You've uncovered the kernel of the software industry- if it works well enough, don't touch it. This is why legacy software hangs around for longer than it should.

You can remove the word software from that and apply it to almost anything. The whole earth rests on legacy temp fixes.

Yep. Just like the old saying, "There's nothing more permanent than a temporary solution."

My feeling is it needs to be communicated non-technically, execs need to think 'what goes wrong if we dont do this', 'how much does it cost'.

THIS is your answer, leoqa. The business value is the $$ the company can lose if it goes wrong.

If higher-ups have no technical insight, round up and give them two numbers: $$ if a bad thing happens, and $$$$$ if the worst thing happens.

And they’ll ask for the probability of these outcomes to find the expected cost. And you’ll struggle to give one without heavy caveats, and thereby lose perceived credibility.

"It's hard, so do nothing" is your advice.

Yep. I recently had to convince management to let us fix a legacy CloudFormation template that was brushing right up against the AWS resource limit for a single stack (which also means it was WAY over the recommended limit for a production stack). It had been causing CFN to occasionally choke on a deployment for over a year, and it was just a matter of time before we either added the resource that pushed us over the hard limit, or CFN choked on a deployment and failed to roll back to the last good state.

The director of my department kept asking, "But do we HAVE to do this now?", so my final answer was this:

"Do we _have_ to? No, but it's like a bridge. Bridges have an expected lifespan when they're built, and they're supposed to be torn down and rebuilt after that. It's not a HARD limit, though. The bridge doesn't immediately collapse the day after its lifespan, but there are fault accumulating. Metal fatigue, erosion of support material, and other things. Commuters can't tell the bridge is past its lifespan, and they'll continue crossing it every day without issue. And then one day the issues will accumulate to the point that the bridge fails and people DIE. That's the situation we're in with this stack. It will continue to function just fine until the day it doesn't, and on that day I won't be able to tell you how quickly we'll be able to recover the application stack, if at all."

That FINALLY got through to him.

> The director of my department kept asking, "But do we HAVE to do this now?",

If someone says that something is a risk and we should address it, I don't expect a director to ask for a specific date the risk to become a disaster. Instead, he should just add it to some todo list and slot some time to analyze it and and budget resources. AKA, doing some "management" work. You shouldn't have to explain his job. You didn't need to learn semiconductors physical properties to know that storage can fail and you need to do backup.

Honestly, in traditional engineering firms it’s not the job of the mechanical engineer to quantify risks monetarily, and it shouldn’t be in IT either. It’s other people’s job to turn "without urgent repairs, we’ll have to put severe tonnage limits on this bridge or close it altogether or risk collapse" to the cost of inaction in terms of economic loss.

I also think it’s fair to expect that non-technical people learn some technical reasoning. They don’t need to be engineers, but IMO it’s completely ridiculous that we just let anyone who is business-adjacent off the hook for not knowing anything.

Business people should understand, at a high technical level, how their products work and what they’re actually asking to implement.

I’ve worked with many great business analysts and product people and consistently the best ones are the ones with the “engineer brain”. They might not know how to read even a single line of code, but they understand software products from their pieces. Those people consistently propose high-quality and feasible solutions. Other business people do not.

As developers, we are expected to constantly think on our feet and adapt. We learn new products and methods on practically a weekly basis. But for whatever reason, the business folk are not even expected to understand the product they help build. There’s such a large disconnect there between dev and business.

Well, you see the problem is that all of these stakeholders are engineering managers. If the director of infrastructure in the engineering organization can’t understand something like “move off static secrets embedded in hundreds of applications to workload identity” as a value proposition then I’m honestly not interested in massaging it further for them.

Write them if they do not mandate basic security right their brandy or vodkas or whatever will be served with opposite temperature they expect...

You really think it’s wrong that you need to show how it will save money in the long run? Why do you care if the software is poorly written? Because you don’t like fixing bugs? Why should they care? If you care then you need to give them a reason to care. I am not saying it doesn’t matter, I am saying you can’t just expect them to agree to do it because you don’t like it. And it seems like that is what you are saying. “Because security” is not an answer. Everything can be made more secure. Show them the risk and impact.

I've been in plenty of meetings with non-technical folk who basically shut their ears off whenever anyone brings up explanations for things like why security matters.

I now work in a highly regulated field (payments/banking) and it's a bit better, but from prior experience if there's no regulator to beat people with, they will simply not give a shit, the only thing they want is to shove out whatever feature as fast as humanly possible.

As an example GDPR was a godsend because I could finally convince people to get rid of the 11 trillion stupid analytics and tracking bullshit we had "Because GDPR", whereas no number of arguments, no matter how well researched and presented, would ever convince anyone before

It never ceases to amaze me how common this situation is, where a quantifiable compromise is clearly the answer between engineering and leadership, yet none is presented by engineering. No party is ever inherently trustworthy, we all need quantifiable reasons to make decisions. I've never unquestionably trusted leadership, so I've never expected them to trust me.

The business value is the savings from the cost of a security issue times its probability

That’s an intractable number that you could cook up but it’s just busy work in my opinion and the premise that everything that is obviously important needs a revenue number is wrong.

You first need to assess the risk of the current setup, and explain that the new setup will reduce the risk (which you may be able to express in monetary terms).

Hire a hacker to exploit the security mess.

You could make an analogy to getting a flu vaccine. There's a small up-front cost, but you're making a bet that it will prevent or reduce the severity of a larger issue down the road. There's no guarantee that the larger issue would ever occur, but "getting the vaccine" makes it less likely to occur, and less costly if it does occur. Skipping the "vaccine" is a reasonable choice if the business is willing to incur substantial damages in the case of a large event in the future. To put hard numbers to it, you could always show numbers like "X% of our customers are likely to leave if we leak all their data to hackers", and base it on documented examples from recent history.

If you can't communicate the value of a proposal then you shouldn't be writing proposals. Find an expert and ask them to do it for you.

I’ve been a staff engineer in FANG for 15 years, presented many times, won awards, done sales pitches etc. I’m not interested in cooking up a number for an executive who can’t use their critical thinking to prioritize. Not everything is reducible and honestly it’s a bad faith take.

This is why open source eventually wins out, it doenst have this pressure.

It seems straightforward to present the tradeoff in a formulation that statisticians can understand, eg. the expected value of an 80% correct solution that ships now is higher than a 100% correct solution that will take 3 months. The opportunity cost of the extra care often only really pays off for security-sensitive code, like crypto, where the cost of failure is potentially catastrophic.

Thank you! I entered the industry a lot more meticulous, test driven development, integration tests, end-to-end.

But shipping faster was pushed at almost every company until, if it's not revenue generating, stop putting resources into it. Not all are like this but it's not all or nothing, it's always a sliding scale how much you can put into making the software correct.

I'm a software artisan deep down but I could never justify putting too much time into making the stuff I used to build "clean and fully maintainable". I tried, but eventually I realized that if I didn't get things out of the door, my company would start losing money and I'd lose my job. Who cares about perfect software in a bankruptcy auction?

There are things you buy out there that don't need Swiss quality because most people will simply not need the quality or won't have the money to pay for it.

I think the biggest problem isn't that a random webpage is sloppy, but when a critical system that could kill people is built the same way as the random webpage.

So what I'm hearing is that software quality will not improve without government regulation.

I can't say I disagree. How it is with engineers they get certified as PEs due to the responsibility they have. In my mind it be similar, you don't need everyone engineer certified, but if you over X revenue or employees side it should apply, and be required to have Y numbers of certified based on company size. Then the companies like with their lawyers, will have to figure out how to follow the regulations and push forward.

I think this is good thinking. I work for a state agency that deals with construction codes and licensing of building contractors, electricians, etc. If the safety of the web is in such a critical state of vulnerability as to potentially impact what the article author terms "normal" non-tech people, maybe there should be some certification that is government-regulated like you suggest. Or maybe that'll happen after real (not potential) devastation occurs - that's how building codes came around.

I don't understand. Do you people just figured out how to write good software? Not long ago, 70% of projects just failed. Not were written badly, but just weren't completed at all. I don't know current number. But looking around me, it's so obvious that we don't know how to write good software. For a reasonable price anyway.

What that PE certification will check? That guy knows how to write unit tests? Well, that's not much. Who will write certification? Where's that genius, why he's hiding from the rest of us?

Maybe building is figured out. I don't know. I still see bridges failing and buildings crushed, so apparently it's not perfect. But I do know that programming is not figured out. Absolutely not. We can apply tremendous effort, pay billions and produce some simple firmware which can be uploaded to a martian rover. But that doesn't scale to the rest of industry.

No, it puts the company at liability, and the PE.

Then when company forces a product to skip safety checks, test units. And the PE proves the did their but the company pushed forward, the company gets wide open to fines and lawsuits.

Make a company responsible and hit their bottom line, other will think twice before they say "just ship it, or I'll hire someone cheaper to do it". And no they don't say that, that's how it's done.

When your paycheck or livelihood is on the line, you tend to slowly sacrifice the principles.

I think your being short sighted here. And a aggressive for no reason.

I hear you, but humans are incredibly bad at context switching like that.

Near the point where things become minorly safety critical, people will blur the line. The point where safety kicks in moves a little further out, and we start blurring the lines again.

There’s a reason the FAA is anal-retentive about things with only very minor safety implications, like the in-flight entertainment system or turning on airplane mode. It helps prevent complacency where it actually really does matter.

You either do it the right way 100% of the time, or I’m not taking bets that you’ll do it right on a particular occasion.

> eventually I realized that if I didn't get things out of the door, my company would start losing money and I'd lose my job.

This is like 90% because most tech is different enough that civil torts don’t exist for them yet, so they don’t have to care.

Eg if Google deletes all your photos they kind of just shrug and you’re out of luck. If I deposited photos at a bank who burned them without notice or approval, they’re on the hook for that.

The law will eventually catch up and acknowledge that bits can have value. Then everyone will have to care again.

You are describing my field - Systems Engineering.

We track EVERY requirement to ensure it is met. Maybe it's a safety issue; maybe it's landscaping approvals (I work in Rail). Regardless, it's bigger $ if not caught up front, and if you start grading requirements on perceived risk, you start the slippery slope to not caring. Example: mark all landscaping requirements as "Not Critical". Then scheduled clearing of the right-of-way is missed. Then pedestrians are at risk with blocked line-of-sight, and trees might drop across the rail.

This must be handled at the government level.

The cost of "losing customer data" is too abstract for a company trying to survive financially. Corners will inevitably be cut.

Unless the costs to be suffered by the company for exposing private or sensitive information are made very clear through legislation.

"Forgive yourself for not knowing earlier what only Time could teach" This quote holds for both individuals and groups alike.

Shipping bad software does not serve the business indefinitely. Application/product security was expensive relative to its perceived value, so it was underfunded nearly everywhere.

2026 is saying "here is the consequence."

Completely agree, but the problem is that the execs don’t understand that. People at the leadership level don’t truly understand or prioritize technical debt. Paying down the debt can be really hard to sell to leadership.

Not sure they don't understand. They're operating in a system where they'd be replaced if they decided to repay all the technical debt.

Don't use the term "technical debt".

Explain those super human hacking AIs like the ones that broke containment and hacked outside sites are coming for your infrastructure too.

All the business people have heard about the Huggingface incident by now (or are living under a rock).

This - 100%.

Just take a look at some of the opensource academic, infrastructure or "passion-projects", where there is no management pressures to "move fast & break-things" and ship-ship-ship. They tend to be very stable, very predictable - and very high quality.

Care to name any?

I've seen software that academics write. It's no better than average. It's often cut-and-paste code, or these days AI slop. Software is a means to an end in academia, rarely a product in itself.

TeX

You read my mind...

I don't think we can rationally extrapolate anything from Knuth's career. His entirely CS presence is a pursuit of perfection unlike, well, anyone.

Anki

> They tend to be very stable, very predictable - and very high quality.

Some of them are, most are not.

Actually, most FOSS stuff is anything but stable or predictable. Instead it is constant treadmill, broken compatibility, and recently, thousand of commits from claude et al.

I did say: "some" - you are correct about "most" (chasing the latest "shiny" pattern, method, infrastructure, etc - constant churn)

I agree... many people seem very purist. But I think it depends on what you're doing. Some things make sense to do fast and cheap. I work helping fundraising, and it's often better to have 95% solution that is delivered on time than a 100% solution that comes out after the campaign has ended for example. Management should warn us ahead of time and give us the time and the space to do it right, sure, but how often is that reality? As long as it's a conscious trade off I think it's fine.

If only it were that simple!

Look at the solo projects from professional devs with nobody overseeing them. Look at what so many of your colleagues submit despite the frustration of your thoughtful team lead and manager. Look at what your so many of your team leads asks for even when they've been granted full discretion. Some select few are great, but most are disturbing if you have any sense of quality or preparedness yourself.

Even for decades before AI, there was lazy naive misguided slop everywhere. During the social coding era of the last 15 years this even amplified itself, streamers and influencers and conference presenters -- professional developers themselves -- would share horrid practices to wide applause. Moving fast and breaking things was what developers were excited about. Rigor was dead. Correctness was a problem for later. Code was presumed best written by a rando on github who'd accumulated a lot of stars rather than by a known responsible hand. Security was a copy and paste job from Stack Overflow, not something to be comprehended and considered.

If only it were a top down problem from outsiders!

But the truth is that the call has been coming from inside the house all along, and the outsiders giving direction have had no problem finding celebrated industry "experts" and "thought leaders" affirming and encouraging all the clowning they could imagine.

Developers could always move fast and do things right but most just didn't have any interest in learning how, and many of those who might have just gave up on bothering.

Thanks for the counterpoint. It's true that being sloppy and taking shortcuts isn't always a calculated business decision and it can be a lack of skills.

Businesses would definitely love great code if they could have it fast and easy. The problem is that they can’t. Developers tend to believe that they could generate perfect code if just they were given enough time. Well I have given my team infinite time. I tried it, do you think they delivered perfect code? Of course not. First of all if you don’t stay really close to everything they do, they will spend all their time on useless stuff like setting up their editor instead of using a readytogo IDE that actually speeds their work up, or testing things that will probably be thrown away a few weeks later when we decide that the strategy we were trying to use didn’t really work, but still require lots of time to “refactor” into the completely different thing we now need. Even when they do build the right thing and focus, more time simply does not equal more quality, it can even get worse at some point as you again get lost and start veering off on a tangent and forget what the business is actually about.

In fact, I have myself spent years on solo projects with the goal of making perfect software. You know, nothing like getting a few users to quickly destroy that idea from your naïve brain. You can’t write perfect code because that is nearly impossible for most of us, no matter how hard and how long you try. Think of it like this: can you beat the greats in your favorite sport or hobby? Probably not, you may be good at a small team level, but at a large city level you are almost certainly mediocre. This goes for your coding skills too. How many great developers out there creating great software we all know and use and love? I know a couple of names, that’s it. Most of us are writing shit enterprise applications everyone who is forced to use hates and that has more bugs than PHP in the early days. And when we try our own thing it’s even more horrible. Show me your repos and let me se if you’re the genius you think you are. Oh of course, you don’t have time to spend on hobby projects since you have a life… great, enjoy that, but don’t pretend you’re able to paint the Mona Lisa just because you spend all day with a brush in your hand striking on a canvas that no one can really see and if they did, it would most likely be a repulsive, incomplete, bug hidden piece of shit.

They will hate you for you tell them truth.

You are assuming that garbage code makes us more productive.

That might be true on the margin, but highly doubt it was ever true once you escape a very shitty local optimum.

you mean industries where software can kill people like pacemakers?

https://ijcionline.com/paper/13/13224ijci12.pdf

Wow, it’s almost like it’s really hard to get everything perfectly right and this is endemic to software, particularly when the language accepts incorrect code.

The only two things we know mitigate this are proofs (eg rust borrow checker, lean, etc) and DST which is basically the equivalent of accelerated time testing which is done in other industries too. Most people don’t even bother with property testing

> Devs aren't clowns who make it this far.

simias referred to the industry and you took it personally.

Why is it whenever someone mentions the shortcomings of software industry, there is always a comment saying "no no no, developers were just following orders!"

> “always been frustrated by the tendency of software devs, and in particular web devs”

Sounds like they’re talking about devs

Because by that rationale all industries are circuses filled with clowns. It's incentives not ineptitude. It's a clumsy take.

Because systemtic problems are often better explained through structural relationships rather than personal failings?

I don't disagree with the point that a lot of software that isn't pressured to be shipped quickly still ends up following bad practices, but I don't think we can ignore that the quality issues in software aren't also found all over in many industries because of profit motives?

He spoke of a clown industry. That means the industry as a whole. That includes, but by far isn't limited to, the devs. Which of course have to operate in an environment, most of them set by their employers. And of course, responsibility scales with the position in the hierarchy. So yes, to a large part it is the management setting deadlines, budgets, staffing and planning allowing for technical soundness and of course safety and security.

But also every developer has a responsibility. At least not to promote any technology which is part of the problem.

> But also every developer has a responsibility. At least not to promote any technology which is part of the problem.

Yeah, but money.

Reply on news.ycombinator.com