Uncomfortable Truths in Software Engineering

Secondly, the obvious counterexample here is MongoDB. Way too hyped-up during its honeymoon, and then almost immediately crapped on by the entire industry (to the point where we've PROBABLY gone too far and are unfair now). When hype waves don't work out, this industry is pretty quick about discarding them.

For at least this one case, many people (including myself) believe it's a success. Whether the tooling for Sorbet or mypy ever reaches that point, there are compelling reasons to believe in the success of the model.

Don't take the above as a statement that C++ is a great language. I'm interested in rust and ADA (just to name a few that keep coming up) which might or might not be better for my needs. I'm a C++ expert though, so I know what the limits are.

I like Typescript. The Typescript enthusiasts are also glib about what a pain it can be to teach. This is not a minor tradeoff.

[0] https://en.wikipedia.org/wiki/Publication_bias

There are edge cases that can be argued all day, but if some large codebase is going to be maintained long term it is difficult for me to see how types could be a handicap. Even if a coder doesn't really engage with the type system "properly", as little as capturing information that was already obvious anyway will reduce the bug count.

There are good reasons to expect typed languages to do well.

This is not about being objectively better but only about preference. So that's not a "truth".

Sure people might prefer static over dynamic, but until we have hard evidence that one is actually better than the other, this is only preference which is heavily influenced by the current moment we are in.

Having said all that, I've worked with both and lean a lot more towards typed languages now (more than I did previously). But this is not an uncomfortable truth, because it's not objectively true.

Additionally for things like static typing, the larger the company the more risk-averse people tend to be within the company so things that give a feeling of safety like type-safety are going to be a much easier sell.

I haven't tried pair programming. It sounds miserable. I figure I'd end up paired with one of those people that can't stand to see you looking for something. You know the type: you're looking through cabinets for something and they demand you tell them what and they demand you go directly to the right place. Sure, I'm looking for something, but I'm also learning where other things are.

If you have two people working with each other, then you have to worry if each person is a fit for the problem ... and if they're a fit for each other. Or you have to worry if they are each partially a fit for that problem such that the two of them count as being fit for the problem (and they still have to be a fit for each other). And you also have to keep an eye on overall productivity. You're billing at 2x the rate, so you had better be seeing some noticeable benefit to justify it.

I've had good experiences and bad experiences pairing. I suspect that people who like it a lot already have a bunch of experience solo developing and are generally able to get along with a large number of people. Also, they're probably solving easy problems with an overabundance of budget.

Note that your comment also identifies a key value of pairing (if you can stomach it). When the model your are building is undocumented/implicit and can only be constructed in your mind by careful solitary thought, then you have a weak system design. But a pairing session can only succeed if you make the model explicit. So pairing has to start with constructing a shared model of the work (hard). It turns out, once you can share a model between two people verbally, turning that knowledge into documentation and/or refactoring is relatively easy.

To the the individual programmer, this can seem like extra (wasteful!) effort. Unfortunately, it also means they are likely to make another inscrutable addition to the pile of Jenga that is clear them in the moment, but will be opaque to all next Monday. Mind you, if your management is running a ticket mill and doesn't value design or documentation, that is the right thing to do.

It basically frontloads the communication / knowledge sharing of the structure and reasoning behind the code, and avoids using patterns that only one person enjoys or understands. Which surely is an important part of working on a team.

gradual typing is an easy way to acquire more tech debt once tickets are closed and no-one comes back to add more types.

Corps like a simple plan->code->test->done without any "return" to code considered "done".

No, because exception handling and the answer to "what type are these data?" are product questions. Hard questions that no one really wants to work out because what if our customer base wants something completely different in 6 months?

This is largely reflected in cloud providers being unwilling/unable to provide account spend caps: what is a sane response once the budget goes over? AWS can't tell you; only you can define it. (This isn't to say "there's nothing AWS can do for you here" -- they could but won't for other reasons).

Python is the largest, common language that can support gradual checking systems being added in over time, and mypy has outlined a great framework for doing just that. But it's not a silver bullet and can't be because the problems are cultural, not mathematic. Either spend the time working out what you want your software to do, or don't. But don't argue "strictness" is "correctness".

For example, guilty-until-proven-innocent (res ipsa loquitur) would make poor IT design and operations a lot more expensive, at the expense of a lot more court cases and a chilling effect on new software. For mature core operations, the price paid by society might be worth it. The IT industry would then be forced to come together and address issues at an industry level, if for no other reason than every player avoiding painful amounts of time and money spent in the civil justice system…which is a lot less fun than even the most torturous code reviews.

My sense is (IANAL) that the liability doctrines for software are stuck in the 1960s with all software viewed as a “best effort” sort of experimental exercise that must be protected from the rigors of normal liability. I think we are past that point for a lot of software components. But who in the software industry dare acknowledge it?

I agree with the sentiment that got me into testing in the first place: software is complicated and hard to modify without breaking something. But, TDD for sure isn’t the answer to that problem.

You definitely get less code written than you would individually, but at the end of it everyone involved understands how the code works and why it conforms to the requirements which keeps your team's general knowledge high and turbocharges training for juniors.

It's also just pleasant to do a genuine team activity and have a laugh with people.

This is all coming from someone who greatly prefers working on their own to doing paring etc btw. It is nothing like the joy of getting your mind spinning at exactly the right speed, becoming one with the problem and smashing out a solution - so don't mentally compare it to that. It's much more like hanging out and brainstorming stupid ideas with your friends on the couch.

But, I often work on features that require the knowledge and input of many people. I lose a lot of time getting blocked on simple things by not knowing what I don't know, figuring out what I'm missing, figuring what to ask, waiting for an answer, trying to implement something based on a brief reply, etc...

The few times I've managed to gather a grop to do something approximating mob programming were incredibly productive. The whole time it was 3 people discussing the main thread of code and 1 person off on their own "fixing up some detail". Who was off on their own changed frequently. We were using VSCode Live Share and Live Share Audio.

Which is why software as a skill in and of itself will go the way of mathematics. And then SE's might actually start doing real engineering.

Agreed. As a tech educator, STOP. I've been teaching people to use technology for decades, and, like with math, a huge stumbling block is "I'm not smart enough for this; only 'tech people' can understand this." And no, not everybody wants to develop algorithms or web pages (or fully wire their house), but most people can learn basics of how things work and look for ways to improve their experience.

Also that analogy doesn't work: most people don't want to wire their own house, but being able to do basic problem-solving on your electricity saves the homeowner time and money AND lets electricians spend their time on bigger problems. Same with taxes, as somebody who isn't an accountant but does do my own taxes: Not everybody will do that, but most people could definitely learn how things like tax brackets work.

This guy is a computer nerd. That's fine, but it doesn't make you INHERENTLY BETTER than people who are color theory nerds or whatever.

I assume that a lot of this comes from the fact that computing is now more than trendy, and being knowledgeable on it makes you feel like king of the hill.

Also lots of people going into computing, knowing the recent history. Strong opinions about things based on the last 10-20 years, choosing their little church. While having next to zero knowledge about previous history (tech or even business in general), nor fundamentals about computing, but they have enough data to flood you with their certainty :)

This is likely (imo) due to the fact that mastery is gained somewhat independently and not subject to any pseudo-objective critical review (beyond 'my device x works!'), and no professional rigors are selecting for different (professional) mindsets. 'Those that don't know that they don't know' and all that.

To amplify, a cognitive worker in an established discipline is buried under domain details and painfully aware of the scope of their field. They specialize, and are certified for that specialization. Their ego is better informed about the realities of their extent of knowledge, and yes, they paid dearly to get schooled and certified for their narrow niche. (So you have the inverse effect of possibly insufferable attitudes of experts regarding novel developments and ideas in their specialization.)

For developers who are ~35 and up, almost 100% true, once you get younger than that you’ll find that there are a large and growing minority of people who are very much in it for the money alone without any particular interest or affinity for computing. Their parents, advisors, etc saw how much money was being made and pushed them towards computing as something more like a high ROI trade.

You see the same done for specialized welding for smart, but not academically smart, kids. It’s hard work to get into, but if you get there it can pay very well, regardless of if you care much about the underlying metallurgic properties that you’re dealing with or not.

This is completely fine either way, but it’s important to realize that we’re not really the professional of child prodigies that it once was.

Of course teachers who understand this also make money, see Miss Excel ( https://www.instagram.com/miss.excel/?hl=en ), who was/is making $100k a day with excel instruction videos on tiktok and instagram. It’ll all about leverage.

I think the bottom line is that everybody should be able to earn a good living wage, regardless of their line of work. It feels broken that one would need to seek a specific job or set of jobs to have a hope of living comfortably.

Anyone can be a teacher. The certification requirements are low. The academic requirements aren't anything more than most kids are getting anyway (a degree in anything will qualify you in most US states). Most teachers need to work so their BATNA is "work someplace else doing the exact same thing for roughly the exact same amount of money." What's more (and I didn't really mention it above) is that you can't tie teaching directly to income. Not that you should be able to, but if you're a quant at an HFT shop you can probably draw a line with a pen from your code to tens of millions of dollars in profit that's there this year that wasn't there last year. Do that year over year and it's hard to argue against you getting paid $800k-1M/yr in cash and bonuses. Helping one of your students read a little better, or finally understand linear algebra, isn't going to increase your personal bottom line at all.

This is one of the main reasons I'm a software engineer and shudder to think about doing anything else for a living, despite not even liking software that much.

> It's uncomfortable because many of us didn't understand this going in.

This is probably not as common as you think. Most of the SWEs I know switched over from other fields because it's an obviously easier gig. Epidemiology, architecture, electrical engineering, biomedical engineering, archaeology, public policy. People are painfully aware of how easy we have it and that's precisely why they're (we're) SWEs, not because of any interest in coding itself.

I think this touches on an uncomfortable point that many people don’t acknowledge: our current system generally* pays people according to how much they contribute to the economy and not by how much they contribute to society.

The scalable nature of software means developers can disproportionately add value to the economy, even while sometimes simultaneously working on projects that could be arguably a net negative for society.

* yes, we can all point to people who get paid handsomely without adding anything substantive, but general rules are meant to be applied, well…generally

The barrier of entry is way too low these days and therefore we have this hazing ritual for an interview process. It's absurd to think that after 12 years in the business, I won't be able to figure out what to do with those requirements. I'm sorry, we need 5 years of Java, not 4.

For those young developers, save up your money over a 10 year period and get out.

Right now the best devops people in New York City are making between $200 and $300 an hour. That's obviously good money, but in the 1990s I had older friends who were making $200 an hour as Oracle consultants. At the top, the nominal pay has remained the same, but adjusting for inflation, that is a 50% paycut.

Likewise, lower down, I know lots of mid-level devs who are happy to make $100 an hour right now, but I also had older friends who were slapping together classic VBA apps with VisualBasic, back around 1999, and they were also making $100 an hour, so, again, it's a 50% pay cut.

But it is absolutely true that pay has been surging since about 2014/2015, so I imagine the software developer pay is about to catch up to the highs of the 1990s.

It's also true that the picture only looks bad when compared to inflation. To your point, most other professions did even worse, so the benefits of being a computer programmer are more obvious if the comparison is to other professions.

All in all, after the peak in 2000, the USA economy had a difficult 15 years.

https://news.ycombinator.com/item?id=29553341

You are also confusing our broadly speaking "professional" jobs with some of those other examples and firefighters have early retirement and DB pensions in the UK

Poor teachers: when I was teaching it was more like 20% of the time, and the other 80% was breaking rocks. Planning, marking, parents, discipline, pastoral care, sports, and straight up babysitting.

That’s a big part of why the pay is so different, in my experience. Teachers have to put in a lot of time and effort to make a diff[erence] and the rest of the work is completely unskilled labor.

For SWEs, the next pull request is only limited by your ability to focus and type.

I’m a full time SWE these days, and enjoying the 10x pay.

There's plenty of suffering on top of that time and money too.

https://web.archive.org/web/20201112012023/http://www.medica...

Software development is heavily skewed towards the young demographics. We just don't think about it when we are young and still in love with computers. In the immortal words quoted by Steve McConnell over 20 years ago: "Wanted: Young, skinny, wirey fellows not over 18. Must be expert riders willing to risk death daily. Orphans preferred. Wages $25 per week."

I don't know about that. Software scales almost infinitely. Impossible to double the number of billable hours or patient load of a cardiologist outside of an herculean effort, but software folks can easily 10x the number of people they bring value to in a short amount of time.

> We didn't spend a ton of $ to go to medical school and 3 years of residency. We didn't go to law school and take the bar. Most of us went to 4 years, or less, of school, and here we are making great money by utilizing the greatest tool of all time: the computer.

Honestly, medical school isn't really the correct example to use because it's not efficient at all.

The big reason medical schooling is so hard is because they have to get a completely unrelated undergrad before even thinking about applying to med school. That means racking up serious debts before even knowing if you are going to get admitted. You can imagine how that fosters a culture of diversity and inclusion and help poor, often non-white, applicants reach medicine when the first step is to get into 6 figures debts with absolutely no guaranteed income.

Other countries with similar health outcomes for their population simply admit freshmen into med school right away.

Residency is something else. It's overworked since the total number of residents is capped by the AMA (also you have to fly-in to your residency interviews on your dime, so much for underrepresented applicants!) and honestly there's a lot of cargo culting. 48 hours shifts with no sleep because it's believed that it's impossible to correctly handover patients from one medical professional to the other (hint, this is a solved problem in a lot of high stress cognitive disciplines outside of healthcare). It's a culture of always working harder instead of working smarter, and the people take pride in that strangely enough.

All it really does is create a moat and allow AMA members to charge more because they made themselves scarce.

> Meanwhile, our friends who are teachers, firefighters, child-care workers, etc. are doing tough work daily for less pay, perhaps simply because they didn't use computers much or didn't like them. And yet, their jobs are ESSENTIAL and we need people to do them even though they're grueling and underpaid.

Firefighters tend to have nice pension funds in a lot of cities.

Teacher's pay is incredibly interesting. Engineers are paid on results (this is even more true at serious tech companies where a significant portion of ones earning will be stock comp) while teachers are paid a government approved rate no matter the results.

I know a lot of parents that would pay serious money to teachers based on results (if their kid can get into Stanford/MIT for instance) if said teachers could show a track record of having previously mentored alumni for instance. Instead, the premium is now captured by real-estate since parents that care will pay extra to get into a good school district.

Ruby's syntax allows for writing things in a pretty expressive style, often not requiring parentheses (for parameters) or curly braces (for object/hash literals), and that allows you to write ruby code in a way that sorta looks like a nice DSL. But it's very much not a DSL, it doesn't have "special syntax", it's just... Ruby. Just because Ruby can be written beautifully doesn't mean Ruby "leaned hard into this".

I don't know about Scala.

My impression is that Ruby libraries have leaned away from DSLs over time. Rails hasnt added a new DSL in a long time, none that I can think of anyway.

Likely for some, but not all cases. Pair programming at my shop had a weird positive multiplier effect on velocity/delivery. It was optional and about half the team did it. Different folks work differently, we aren't all productive in the same ways.

You don't need to cover all bases, in 99% of software. (And in the 1% where you do, you should probably be using formal verification.)

Testing is an investment and should be analyzed in terms of ROI. There is a cost to writing tests in advance, a cost to fixing bugs after writing the code, and a cost to shipping software with bugs left unfixed on rare codepaths. Sometimes TDD yields high returns indeed (usually for "library code" that (a) does something very complicated on a complex data structure; (b) is easily isolated from the rest of the system without writing tons of mocking infrastructure; (c) has a stable API that is not frequently refactored.) Often it does not. I've found that a combination of static assertions, black-box systems testing, and incrementally adding regression tests during debugging allows me to develop good code more efficiently than TDD the vast majority of the time. Your mileage may vary, of course.

If it’s easy to do I’ll do it, but none of those are trivial things.

"Capitalism" has about as much to do with software performance as actual devil worship had to do with my Dungeons & Dragons or heavy metal cassettes.

And if your life as an engineer or user is "miserable" today, then I wish I could transport YOU to the 1980's. Yeah, existence is suffering... it sucks to be in the top 5% of household incomes, be inundated with open source platforms and tools that didn't exist too long ago, and a have a supercomputer in my jeans pocket.

Fostering co-operative communities has less to do with the political/control system, and more to do with how it is implemented - and the mindset of the population. I could just as well blame multi-culturalism, or globalism, or lack of religion, or metropolitan lifestyle on lack of social cohesion.