Comment by zer00eyz | Hacker News Reader

Comment by zer00eyz | Hacker News Reader

Parent Full thread

> I couldn't understand why friends weren't interested in what I was working on, until I realized they were building CRUD apps all day.

As someone who gets paid to clean up the cluster fuck that can be created by people building "crud" apps let me tell you better debugging tools would be a godsend.

In the old days with more monolithic apps you at least had one pile everyone shit in, and it was possible to get a handle on how it "functioned" and what its quirks were. Between micro-services and docker it isn't one big shit pile its a bunch of small ones, everywhere. And because of docker on occasion I now get to dig down to the underlying OS, because someone who had no right touching a production server, ever, built and tweaked a container and it "runs" so it went out. Did you know that you can run diff over ssh to compare and contrast the code on two containers? I can think of two rather heathy pay days in the last year where some Jr. on site engineer is the one who found the problem by going bottom up while I went top down.

The article states that "developers suck" and on a bad day I would say that the author is giving them too much credit. On a good day I realize that the issue is much deeper and that we hire technological magpies who want to chase shiny new objects (the javascript world is probably the worst at this). On a really bad day I realize that our entire industry is "wrong headed" starting from hiring. We test people on "can they write code" but no one ever asks the people walking through the door to read 50 pages of it. To find the one misplaced semicolon just to see if they picked up on any of the logic traps that were littered all over the other 49 pages.

So to you sir (or madam), I raise a glass for at least trying to build tools for me, the guy who fixes the "gnarly bugs" in crud apps. I wish you would go back and finish your work, the 6 of us who really appreciate you would happily buy you a beer.

The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in. It's a hard career and industry for this reason. We all know the systems we work on are full of subpar choices, design flaws. We all know the delta between what it is an what it could be. But almost nobody can reconcile that gap on their own. And multiply that by entire teams. It's orders of magnitude less likely an entire team is that skilled and enthusiastic.

I have three philosophical perspectives that help me better frame the software industry.

Number one: People aren't stupid. Rather, things are hard.

Number two: Writing software is like doing a trillion piece jigsaw puzzle.

Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.

Number 4: there's never enough time.

People who could do better are pushed to ship a feature tomorrow because hey it's simple and it can't take more than a day or so, right? And then we have the next feature..

I haven't seen anyone pull off a large code base without ever going back and reiterating on the design and refactoring things as needed in an attempt to try and maintain the qualities we (developers) would want in software. Unfortunately there are too many places in the commercial world where the time for that work is never given.

"Never enough time" is an artificial construct. Set up a pace that gives you a manageable output, nobody will be able to measure you precisely anyways.

Then suddenly you learn to do the right thing ("right" means the one that is not stressing you out) faster. And you're a happy person.

How many managers have you heard giving importance to refactoring or maintenance? (Some pay lip service, but very few actually make sure that its gets done in my experience).

> How many managers

Or customers, for that matter. Sucks when there's weeks' worth of reworking guts, refactoring, and testing to be done and you need to bill someone for the time.

I don't think that they are the experts in this kind of trade-offs.

In my experience the people who have said "I didn't have enough time" if they were given infinite time, wouldn't have done any better a job.

Funny, because aside from some cheap tools and coffee, time is pretty much the only thing a developer needs to do anything.

What I've experienced is that in open source projects, patches get rejected and rejected until everyone agrees they're good enough, and sometimes that takes months or years of iteration. Not good enough? Spend more time on it. That's how things improve.

Wanna write a test suite? Need time.

Wanna run a battery of static analysis tools and inspect their output? Need time.

Wanna run a fuzzer? Need time.

Wanna document things? Need time.

Wanna rework the buggy guts of your application? Need time.

Want to figure out the heisenbug that rears its head every once in a while? Need time.

Realize the API the guy before you wrote is crap and needs to be rewritten, and all components that depend on it need to be updated to match the new API? Need time.

Want to formally verify that the core mechanisms of your critical application are correct? Need time.

Need to figure out and rewrite the part of the program that slows everything down to a miserable crawl? Need time.

On the projects I'm on right now, I could think of months of stuff to do to get them to a state where I'm somewhat happy with them.

Every goddamn thing just takes time and not much else. And IME the commercial world presents the same problem over and over again, far too often: you don't have time, you can't bill your customer for all the work that should be done (if good software were a concern), and you can't block the thing they want just because you first want to fix three other things that'd take a while to do. Crap just accumulates, because people build in a hurry, on a bad foundation.

Even open source projects where deadlines and "we sold this as an item that takes three days of work" aren't a thing, time is always a problem because there's only so many people putting hours in.

"Even systems with well-defined architectures are prone to structural erosion. The relentless onslaught of changing requirements that any successful system attracts can gradually undermine its structure. Systems that were once tidy become overgrown as PIECEMEAL GROWTH gradually allows elements of the system to sprawl in an uncontrolled fashion." http://www.laputan.org/mud/

There is that old adage "perfect is the enemy of the good", well I would argue that "working is the enemy of the good" as well. When your "average" engineer spends 2 to 3 years at a company how often do they have to pick up the messes they make? Rarely? Never? It doesn't pay to be a good citizen in that case - it pays to be (as another poster pointed out) fast.

> The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in.

If software is hard, and average people are not ~willing~ to put in the work to make it less so, doesn't that make them lazy? Willfully and willingly lazy at that.

> Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.

Look around you,how many engineers do you know who's code you would be willing to bet your life on? The problem is that every engineer thinks they are a surgeon (and a good one at that) when many of them wouldn't even make good nurses.

It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.

No, it simply makes good systems brutally improbable.

Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.

> It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.

If I walk the 1/4 of a mile flat loop at the park down the street from my house every day and call my self a hiker I would be a liar.

Really were talking about Dunning–Kruger, "If you're incompetent, you can't know you're incompetent". I go back to "stupid" as the clear winner, and add in that they are too lazy to get out of that rut (put in the work) and that the title "engineer" becomes a willful or delusional lie.

> Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.

To your earlier point, we give everyone with a pre med degree a scalpel and a patient with a traumatic brain injury and say "go" were going to have a lot of bodies. The case is clear that we are far too tolerant of bad code and bad engineers, and we don't have roles like "podiatry" or "nurse" to demote people down to where they can contribute and we can limit their harm.

In healthcare, podiatry and nursing aren't demotions. They are entirely separate career tracks from medicine. Most nurses and podiatrists do those jobs because they want to, not because they failed at becoming physicians.

"The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in."

It's not just that, but the typical company is too busy chasing new features and fighting fires with understaffed engineering teams on constant death marches to unrealistic deadlines. Nobody has time to do things right, though many of us want to. Instead the systems accrete endless layers of bandaids and half-assed solutions which everybody hates but no one has the luxury to fix or solve properly the first time.

You need strong engineering management to communicate the business advantages of well-made software. The benefits of good software are exceedingly hard to measure and understand as a businessperson, because design is a lot like risk management: you can't measure the cost of the disasters that don't happen. The only substitute is to communicate thoroughly on the business's own terms and to show examples of payoffs whenever they're tangible enough to make the point.

Number 3: MOst SOftware Engineers are bad at their job, they aren't anything close to Surgeons.

This I think is the reason most software nowadays is garbage, there is a lot of low skilled people creating it. I blame bootcamps and dumbed down CS courses for this. It's not necessarily the developers fault, they can't improve on the things they don't know they are bad at.

Computer Science curriculum was never intended to produce competent software developers. You're complaining about the wrong problem.

Out of curiosity, do you consider yourself the exception or the rule?

> Did you know that you can run diff over ssh to compare and contrast the code on two containers

Good luck doing that with Windows containers and diffing the HKEY_CURRENT_USER hive.

Anyways, rollback the container and start over.

I haven't touched a production windows box in 10 years, and probably haven't run windows on the desktop for 15. Your comment did access enough of my windows PTSD to make me twitch. I have abused the ldap interface for active directory several times in those same years periods but I don't think that really counts.

The situation of the "live" (container) works and we can't figure out how to build a new one comes up more often than you think. Knowledge leaves the building. People run off in the weeds and then quit so they don't have to clean up the mess they just made and no one is willing to touch it. Some times an executive shits a kitten and your policy changes from "vendor everything" to "always use the latest and greatest version cause "security" . Sometimes in house staff are stupid, or incompetent, or lazy, or is smart enough to know that if they touch it they own it forever and choose to play one of the aforementioned roles. Sometimes the staff does know and no one in management listens to that guy or gal. Other times it is a minefield that every one dodges on their way to getting what ever else they need done, and no one addressed the elephant in the room till it causes a panic.

I think there are more versions and variations of the "we don't know how any more" tale that I haven't listed, and I am sure that my list is no where near complete however I'm sure you get the picture. The call aways starts the same, I always listen and nod and say not a problem, and use soothing tones to tell whoever is on the other end of the phone that the world isn't going to end. Some of them freak out when I ask for an NDA, and a development environment - many give in but some are unwilling to let me peek under the kimono. Most of the time I can tell them that "price" might get better (or worse) but it will be more accurate if they do so. It is good to get into a system, to talk to someone who can "solve your setup problems" because most of the time they will point you to a closet marked "dead body(es)" and in many cases your half way there. Sometimes it is simple and large (back to that elegant) other times there are a few corpses or an orderly grave yard, other times it is an Alderan level apocalypse and you shut the door and slowly back away.

At that point I can give a rate, and a guess at a time line for a fix (and yes I estimate like scotty from Star Trek). Most of the time the person on the other end of the phone is desperate enough to say yes. With that magic little word the hunt begins - and many systems are shockingly resilient to being ripped apart and abused. Often times things that look important can be ripped out, swapped out or shut off and things still happily churn along up to a point. From a global view you can quickly get to a city or block sized area and then the fun begins. The solution in more than half the cases is fairly simple, I'm changing code because requirements, directions and documentation artifacts have become more sparse as time goes on. Sometimes I'm quashing good ideas for the sake of consistency but most of the time its some sort of "cute" piece of code that is getting ripped out for the betterment of all involved.

Rare is it that I tell someone to really "start over", to burn the whole thing to the ground and not look back. Those closets with Alderan in them do exist and are frighting and fun when you find them. I do love a green field but I am wary of the company that offers me one because most of them make a whole separate set of mistakes on V2.

No one every reads any of the 475 billion lines of crap they are updating, who really knows if you are making things worse or better, or where any of it comes from. Most of us wouldn't eat a ham sandwich handed to us on the street by a stranger but will happily run code from 100's of other people on our servers with out even looking at it.

Reply on news.ycombinator.com