Hacker News Reader

Comment by airza | Hacker News Reader

I feel like the slow decline of software quality has been in lockstep with the gradual transition from (expensive and non-measurable) manual software/hardware testing and QA to automated frameworks and rollout-based quality assurance.

I constantly encounter broken functionality, buggy or unpleasant UIs, just as the author has. It feels like many of these problems could be avoided if you just had one person whose job it was to sit there and look for broken stuff. (I'm sure I'm biased as someone whose first job out of college was to sit there and look for broken stuff.)

View on HN

I would tell a slightly different version of this story, focusing in on "rollout-based quality assurance".

I would say that effortless, automatic updates are to blame.

When you can always just push an update, the impact of a given bug goes way down. It's no longer mission-critical to exterminate flaws before shipping; a totally broken feature becomes a mere annoyance. So project prioritization shifts from polishing an artifact to outweighing the (presumed inevitable) constant stream of little annoyances with fixes and features. I think the shift towards automated testing is just a symptom; an attempt to bridge the gap in this brave new world.

For a clear-cut example of this phenomenon, look to the video game industry. Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media.

Right around that time, "glitches" went from very rare unicorns that people would spend lots of time actually seeking out, to nearly everyday occurrences. As long as it doesn't corrupt someone's save file, they mostly laugh it off and upload a clip to YouTube to show their friends. This is just how things are now.

(Edit: I should have scoped this to "console games")

> Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media.

Sure, but they still (sometimes) released (a few) extra revisions of a game. They were just targeted at people who bought physical copies after the revision date, rather than at existing customers.

Or said updates came on the 1.0 version of the game as shipped in markets that got the game later than others. (Just imagine — per-market release versioning. Every market effectively got its own fork of the codebase!)

Or said updates came in the form of a re-release port. There are patches made to the emulated downloadable app-store re-releases of some games, that never made it into any physical edition of the game.

Also, before home consoles, arcade game machines did receive bug-fix updates regularly. Arcade machines were essentially provided “as a service” from their manufacturers, with support contracts et al. Sort of like vending machines are today. If you reported a bug, they’d fix it and send you a new EEPROM chip to swap out in your cabinet. If there was a critical bug that affected all units, they’d send techs out to everybody’s machines to swap out the ROM for the newest revision. (For this reason, it’s actually kind of hard to do art-conservation / archiving of arcade games. The cabinets almost never have fully “original-release” components inside.)

I'm sure this happened occasionally, but it wasn't advertised. Nobody was buying a new copy to get an update. Reviewers weren't revising their reviews in accordance (something which does actually happen now). It was still absolutely mission-critical to get things as polished as possible the first time around.

I agree that games used to have far fewer bugs when shipped, but it's not true that games never received updates back in the day. I distinctly remember queuing on file sharing sites as a kid in the early 2000s to download half-life updates and updates for other games.

I suppose my statements above should be mostly limited to games on consoles, not PC. That's what I had in mind.

I remember when DLC used to be called “a patch” and it was both bug fixes and also huge amounts of new content. I wish I could remember what games this pertained to.

I think you've both got a piece of it. I've programmed PC software, embedded software, and mobile software, and my gut feeling (without data) is that the shipped software quality is inversely proportional to the update frequency and ease of updates. Had nothing to do with how smart or skilled the developers and testers were. Had nothing to do with management's priorities. Update frequency and ease changed immensely once we could feasibly deliver patches over the Internet. Before easy updates, you'd actually quality check every corner of the application, you'd actually fix those P2s and P3s. You'd do exploratory testing off the test plan rails to find things. There was even a concept of "done" in software, as in, you eventually stop constantly jamming features in and tweaking the UI in maddening ways.

Now, it's just "LOL just ship it, users will just deal with it until the next release!" Now, it's "Do experiments on N% in prod and use end users for A/B testing. If something's broken we'll update!"

In several industries, it's actually totally expected that v1.0 of the application simply won't work at all. It's more important for these companies to ship non-working software than to miss the deadline and ship something that works! Because who cares? Users will bear the cost and update.

I have witnessed this first hand comparing two systems.

One system has no patching, and updates incur some non-trivial amount of effort on the part of the installer. Releases are a few times a year, at most.

The other system has patching, updates are lighter weight, and as a result, the system has THOUSANDS of patches released over the last decade, north of 2 per work day.

Guess which system is higher quality? The former.

Much, much higher quality.

absolutely this. Back in the early 2000s quality assurance was insane. A release is was working on (aaa title from a major studio) was blocked because players' eyes were rendered incorrectly and you had to really zoom in to even see the glitch. And of course you had to find and fix all bugs before october, or else you won't be able to hit Christmas sales.

Once internet updates became the norm, it all became pretty much like the rest of software industry. (At least game companies still have QA departments, a lot of mainstream web companies have dispensed with those as well.)

Or as I tell my wife. I am playing my favorite game 'updating playstation'. I turn the thing on so rarely by the time I usually do turn it on there is 1-2GB of updates waiting. Glad I have a semi decent internet connection these days...

> Until around 2007, games received no updates.

Warcraft II, from 1995 received multiple patches. So did many other games from that era.

Do you perhaps mean console games when you say "games"?

Um, how long have you been in the industry?

Software has always sucked and had these issues. It has nothing to to with automated QA. The reason you see more issues is 'way back in the day' your software had a very limited number of things that it did, and in general it did not involve accessing a network or chugging down massive volumes of data from untrusted sources.

I work for a company that has a lot of individuals that test for QA issues, they have list miles long of things to check and write reports on.

The problem is more of "It's much easier to write mountains of code than it is to ensure that it works in all cases"

Software has never been better; there are just more users to please.

And despite the issues lamented in this think piece, it is _Apple_ that the author should blame for setting technology expectations impossibly high.

No organization had been remotely as successful in understanding and releasing tech products that were truly great.

Everything since is just a comparison to expectations Apple set. Even when Apple fails, it is in comparison with an Apple that does not.

The one thing that has changed is updates can be delivered easily. This takes some of the pressure off in terms of QA because rolling out a fix to a centralised service delivered through the browser is quick and painless. The cost of pressing millions of CD's kept developers in check in the past.

That's really not the only thing that changed.

Games are also massively more complicated today. It's one thing for three people getting a 2.5 MB single player DOS game reasonably bug free. It's an entirely different thing to get the 5 GB game made by a team of 100 or 1,000 people.

Most of the size difference is assets anyway, right? Games today use off the shelf engines. It should be alot easier to make rubust games when most of the techincally hard parts are allready done.

> Software has always sucked and had these issues.

I disagree, subjectively it had its ups and downs and we are in a down phase right now. YMMV.

And, as you’ve alluded to, the scope of what software does for us on a daily basis has expanded by several orders of magnitude. Not only have a number of devices that used to be purely electro-mechanical been reworked to use microcontrollers (cars, everything in the kitchen), but the scope of activities that have migrated onto the web or our phones is truly massive.

30 years ago, software bugs might interfere with you professionally, but they wouldn’t stop your ability to get money from the bank, cook food, or do any other day to day tasks.

> Not only have a number of devices that used to be purely electro-mechanical been reworked to use microcontrollers (cars, everything in the kitchen),

Yup, and in most cases this not only did not improve them, but made them less useful and more fragile. Let's be honest: the software is there only because it can save on manufacturing costs, and sometimes can be used for extra marketing benefit. No attention is being given to providing value to the customer.

I have to disagree in the case of car engines. When I was a kid my dad and uncles spent hours each month fixing minor problems with their purely electro-mechanical cars. I don't miss carburetors or mechanical ignition timing one bit. Electronically controlled engine functions are much better both in terms of efficiency and consistency.

That's true. I don't have that much experience with car repair so I might be wrong in perceiving the 90s and the 2000s models as the optimum in terms of car reliability - the stuff mostly works without funky issues, parts are cheap, repairs can be made by anyone who spent some time with a wrench, and you don't have to visit a shop with a license for poking around car's computer over every minor issue. That is to say: advances in computer control don't have to go hand in hand with making the cars expensive to service and not end-user repairable. But they do, because greed.

Depends on what you’re talking about.

Car engines are vastly improved in both reliability, cleanliness, and efficiency by the introduction of computers into them. You might not like that when it goes wrong, but we all appreciate not breathing in pre-computerized car engine exhaust.

And that’s the rub. While shoddily written software shoehorned into cheap consumer goods obviously degrades the experience, there are tons of places where well written software has massively improved the quality of the goods that they’re added into. Objectively car engines are just better for the addition of software both in design and in operation. They’re smaller, more powerful, more reliable, cleaner burning, and more efficient than they were before we computerized them.

Yes but at the same time the auto makers let another team make Electron apps for the dashboard. Engine ECUs are nice and decoupled. Maybe the hard realtime requirements is what saves them and keep all the novell crap out?

And that’s why this is hard. There are cases where software drastically improves the objective quality of things (car engines), cases where well done software makes items significantly more enjoyable (some car infotainment), and cases where software makes things worse (everything in the kitchen). Separating them out is hard.

I worked for almost 15 years in embedded and have 25YOE, and no, software has not always sucked as much as it does now.

I agree with your last point a lot, though I would modify it slightly: it's much easier to write mountains of code now than it as, and it's now common and much easier to import external dependencies (especially at system level) than ever, and those dependencies have tens of millions of lines of mediocre code all by themselves.

You also need management willing to prioritize engineering time to fixing that broken stuff, and engineers who actually know how to make non-broken stuff. My experience is that having all three of these prerequisites is pretty rare.

I'm going to restate something that I said a few days ago:

There are a number of hats developers are expected to wear today:

1. Developer of new features

2. Sustainer of prior code and features

3. Tester of all of this

4. Constant student (outside work because who'd pay their employees to learn?)

The priority for the business is (1), so 2-4 get neglected. This compounds over time to mean that old code isn't properly refactored or rewritten when it should be, and none of the code is tested as thoroughly as it should be, and none but the smartest or most dedicated are really going to be perpetual students (or they'll choose to study things that interest them but don't help at work, like me).

When the old code and poor tests create sufficient problems, you get a business failure or a total rewrite. Which strips out half (or more) of the features and the whole process gets restarted.

Don't forget:

5. The mentor of younger developers.

It's another thing some companies expect you to do, but don't allocate time for it, so at the point you finally know enough to not do a bad job, you're suddenly being pulled out of 1-4 and expected to do 5.

Oof, we had a guy retire over that. In his appraisal he was dinged for not doing enough. They counted his training of new hires as 1 "point" (or whatever, the system was weird). They neglected to consider that he was training 5+ new hires in that year. He'd had enough, put in one last year to wrap some stuff up and then he was out and free.

And also lots of devops likely these days:

- Infrastructure design

- Deployment

- Monitoring / log-based debugging and fault tracing / handling customer issues

In addition to wanting a "full stack" developer of course...

Right. It's the curse of the name. DevOps wasn't meant to be a role, but a philosophy of working and organization. But management (and practitioners) latched onto the idea of it as a role and hosed themselves. Now you've got a 20+ year veteran developer tasked with keeping dozens of servers properly configured, secured, and operating as well.

At some point we have to accept that specialization isn't just for insects. It's helpful to have a proper sysadmin or DBA or whatever (appropriate to your domain) within the team, and not just diffuse those roles amongst the developers themselves.

As someone who has been programming since 1984, software quality has greatly improved not declined.

The level of complexity in modern day software is orders of magnitude greater than that of even a decade ago.

What has changed is our reliance on that software. We are now so deeply embedded into our software existence we see these flaws up close.

Great comment. People also used to use only a few pieces of software in any given day (or week, even!) e.g. email, web browser, and word processor.

Now, our computer ("phone") is with us everywhere we go and we'll use dozens of complex applications per day, connected by dozens of APIs, networks, protocols, and hardware features. It's a miracle any of it works sometimes! Thank you to everyone for making this stuff seem like magic; my twelve-year-old self would be amazed at how well it works.

I do feel like there are more UI bugs as we optimize for certain metrics over others. Timing updates has become far more complicated, so we get weird UI refreshes as new data comes in, stale caches, missed notifications, etc. Turning it off and on again often works, surprisingly (probably because devs start with clean environments often, so that's the functional baseline).

Lastly, it is probably far more lucrative for a technology based business to use their most valuable minds for the Next Thing, rather than iterating on the current thing. Incremental revenue improvements just don't cut it in a capital-driven world; everyone is trying to escape the local maxima to find billion/trillion dollar businesses.

Yeah, but then you'd have to raise the price to pay for the testing and the jokers a block down the street who YOLO'd their competing product into the marketplace without testing would grab all your sales. You'd be out of business and your competitors would be laughing their way to the bank while the customers still suffered constantly from a broken product.

If it were only for novel products this would be a reasonable argument. See Netscape's rush to release their browser as an example of what you're talk about.

But the worst part is that this is an issue with established products that have secured their market, and will even receive money every year from their customers. They have both the money and the time to pace themselves and test things properly, but they don't.

Can you name a few of those who really truly have time but don't take their time? Also it seems there's always an opportunity cost. So time is always "of the essence".

Right, you could be spending those dollars running ads, which the market is more likely to respond to :/

Microsoft, Apple, Google.

An MS bug in an enterprise piece of software (so they get paid every year for it) is Skype for Business. Perhaps my org hasn't updated their installation, but if I drag a contact from the chat window (say you message me and aren't in my contacts yet) and then drag it to the contact list (in status view) it will reliably crash the program. If I drag it to groups view it'll place the contact in a group. My guess is that dragging it to status there is no "default" group and so there's some kind of null pointer exception occurring (it tries to add it but with group given as either junk or null).

I used to see more bugs in Outlook, but it seems somewhat better with the last update so I haven't noticed them. Though it was fun when I had negative 2 billion messages for a week or so (I certainly didn't have enough to cause overflow so I have no idea how it wrapped around like that).

Google's is an issue of usability of their webapps (IME), not strictly buggy but not sufficiently tested. Behind the proxy at work Maps is incredibly unreliable. It takes several reloads for it to actually start working "correctly", but don't change where you're looking too much (you can zoom in, but do not pan around). That's not the only unreliable one in this situation, but it is the most easily demonstrated.

Apple's Messages and Mail constantly tell me (they've been better the last few months, but it still happens) that I have unread messages, I'd search and search and never find them. Then I'd pull it up on a different device and finally see the unread message (which was both visible and marked as read on the original device).

Some of these may be shallow or seem petty, but it's an unpleasant experience that after so many years and with so much money should've been resolved for each of them. I'm willing to tolerate an indie game crashing on me. I'm not willing to tolerate an enterprise software solution (MS) crashing for a natural user behavior.

EDIT: I think resolved, but Apple's iOS calculator bugs were annoying for such a simple program. Not strictly a bug, but Windows' calculator, these days, is an unusable mess in many ways. It shouldn't require so many system resources to add some numbers together (the same could be said of many small utilities that were rewritten for, I think, Windows 10 or Windows 8).

Gmail showing me email meant for <first><last>@gmail.com instead of <first>.<last>@gmail.com. The fact that they ignored the . at all in the user names. The Youtube app on iOS when I first installed it and quickly uninstalled it years ago was an incredible annoyance. It wouldn't reliably show me the video I'd actually clicked on that caused the app to open.

Adobe is an almost perfect example of holding their market captive. I've run into a number of Adobe Acrobat issues over the years, though fewer recently (but I use it less often now). But especially Acrobat Reader on Mac OS X was awful, I actually once had to reinstall the OS to get it to stop fucking up PDF display even after uninstalling the software (I never found out what it had done to the system, and gave up). I needed it because I couldn't find anything else (at the time) that supported digital signatures in PDFs on the Mac (in the sense that it actually worked, I think Preview let me do it but what it made wasn't usable by the people receiving the file).

EDIT2: Another Google one, with Chrome on macOS in full screen. Hiding the location bar means you straight up can't get to it. You have to reenable it, versus a sane behavior like autohiding and moving the cursor to the top restoring it.

Skype for Business is just Lync [used to be Office Communicator, maybe?] renamed/reskinned, right?

I think that product is cursed. It was always a dumpster fire, like IBM's Lotus Notes. It's there so you their clients don't accidentally try Slack or any of the sane alternatives.

All products of MS, Apple, Google are in a constant churn mode. New design. New integrations with whatever platform changes happened in the back, new features to match the competitors, new trends, new mobile apps, new browser features, new framework.

Those are the products that are not the real products and not real money cows, so they get very limited attention.

I agree all of these are horrible. GMail is still a slow piece of shit. I recently tried Thunderbird, and .. it slowed down too. Wtf. Slack is slow too. Typing has become slow for some reason in a lot of "apps", maybe too fancy fonts?

Anyway, these companies have huge opportunity costs. Just look at Google. They try whatever crosses their mind and nothing is good enough compared to "ads". And so they shut them down because opportunity costs. (Not because upkeep, but because then your attention is not on the next big thing, whatever that will be.)

> I agree all of these are horrible. GMail is still a slow piece of shit. I recently tried Thunderbird, and .. it slowed down too. Wtf. Slack is slow too. Typing has become slow for some reason in a lot of "apps", maybe too fancy fonts?

Possibly because more companies are moving desktop apps to Electron so they can run JS everywhere.

So more telemetry is not a guarantee for bugs being fixed.

> They have both the money and the time to pace themselves and test things properly, but they don't.

But not the motivation, because "it works" and improving UX would cost money and doesn't have an immediately visible return.

I agree with this sentiment in general. Of course, we have a ton more features than we used to have, but I think given the newness of software in years past, we were OK with bugs and issues because of the novelty of it all.

Now we are in 2020 and iPhone updates STILL cause battery issues. The iPhone has been out for 14 years...

Our expectations have changed. Tools like Excel should just work - and yet, when I try to save a file, sometimes it freezes and crashes. How is that acceptable now?

This, but also remember that we're living through a transition from SAAP to SAAS, where the "service" is actually data extraction for advertisers. For that, software must be minimally useful to a user, but the true aim of UX is not user-satisfaction but data extraction or subscription lock-in.

I work somewhere that primarily relies on manual testing rather than automated testing. It definitely does not make the software more reliable here, at the least. :-)

There're three kinds of manual testing:

1. Manual testing that should be manual (exploratory).

2. Manual tests that are new and haven't been automated yet (but will be).

3. Manual tests that should be automated.

(3) is the one many people see and suffer through (I know I have). They need to be automated to free up time for (1), which is where many issues are actually discovered. But if (3) dominates your time, you can never get to (1) and you'll constantly ship broken things (or more broken than they should be).

This is very insightful, thanks!

Reply on news.ycombinator.com