Beginner users are also very good because they do stuff you never expected.
Beginner users are also very good because they do stuff you never expected.
People using software for a while undergo operant conditioning: consciously or subconsciously they quickly learn to internalise what makes the software 'angry', and learn to avoid that. That makes them more productive users, but also less useful as explorers.
See https://blog.regehr.org/archives/861
> Have you ever used a new program or system and found it to be obnoxiously buggy, but then after a while you didn’t notice the bugs anymore? If so, then congratulations: you have been trained by the computer to avoid some of its problems. For example, I used to have a laptop that would lock up to the point where the battery needed to be removed when I scrolled down a web page for too long (I’m guessing the video driver’s logic for handling a full command queue was defective). Messing with the driver version did not solve the problem and I soon learned to take little breaks when scrolling down a long web page. To this day I occasionally feel a twinge of guilt or fear when rapidly scrolling a web page.
> The extent to which we technical people have become conditioned by computers became apparent to me when one of my kids, probably three years old at the time, sat down at a Windows machine and within minutes rendered the GUI unresponsive. Even after watching which keys he pressed, I was unable to reproduce this behavior, at least partially because decades of training in how to use a computer have made it very hard for me to use one in such an inappropriate fashion. By now, this child (at 8 years old) has been brought into the fold: like millions of other people he can use a Windows machine for hours at a time without killing it.
Sadly my efforts to promote this practice were met with a dismissive response: “QA leads to lower quality, according to this study”. I just didn’t have the energy to try to change this person’s opinion.
I have seen exploratory testing in concert with good developer testing work extremely well. Old school SQA isn’t really my thing, but modern exploratory testers are tremendously valuable.
Was this produced by the same people who published a study claiming that flossing your teeth is bad for you, or the study that claimed that wearing a bicycle helmet is dangerous?
There are several anti-patterns that sometimes fit under this vagueness:
- Throw garbage over the fence. I.e. low quality products thrown over the fence to a QA team.
- Over-reliance on manual testing.
- Quality is somebody else's problem (sort of related to the first anti-pattern but not exactly the same).
Even the term "Quality" is usually too vague to make sense of. What is quality exactly? How do you tell if your product is higher or lower quality than the competing product? Organizations tend to declare they're the best, regardless of practices and with no real benchmark.
It's a bit like the bike helmet controversy. The anti-helmet people from Netherlands all claim that helmets somehow force cyclists to ride faster or more dangerously, or force car drivers to take greater risks around them. (Incidentally, this is very similar to the anti-seatbelt arguments I heard decades ago.) The helmet isn't the problem here though, even if there's any truth to these claims: it's the humans. The helmet will absolutely protect your skull better when it impacts concrete than no helmet.
But yeah, point taken on the bike helmet arguments, though I think that's a little more clear cut.
I think QA (manual software testing by professional testers) has an important role in producing quality software but there are organizational corner cases where the idea of "QA" in software promotes the wrong thinking/approach to quality that results in an overall poor outcome. Quality in software starts before a single line of code is written and has to be in mind throughout the development process so you shouldn't fall into a trap that somehow slapping "Quality Assurance" on top of bad engineering can create quality after the fact.
However, having a team that is responsible for Quality can absolutely lower the quality of a product. For starters that team is now, at least in terms of perception and more importantly organisational politics, responsible for quality, so by definition others are not held as responsible. That can have all sorts of perverse side-effects.
Secondly, if the QA team is not very good, then it's just a time sink. Misreported defects, queries about functionality, increased communication, supposed environmental issues all generated spuriously, then take away time from development and other teams. Lost time is not replaceable, and limited time drives down quality.
Lastly, having a separate team lengthens feedback and issues are spotted later in the cycle, this drives quality down too.
My preference is always to keep testing within the team, automate it and make the developers think about and appreciate good testing. Good testers, can absolutely contribute, and exploratory testing can be highly valuable, particularly for systems users directly interact with, but they are, in my experience, pretty rare.
- an over-reliance on low quality and fragile tests at the highest and least flexible level of the stack. Usually a GUI or browser based tests.
- under investment in domain synthesis and learning, and an over-investment in either automated testing or robotically following test plans.
- because we've shifted the responsibility and accountability to a different team, the developers may assume that they have a reduced burden to test their work. And you get the typical "throw it over the fence" pattern, with its associated long latencies.
- an us vs them mentality, separate management, and isolated silos of communication.
The pattern I've advocated for is more like: having an experienced and talented exploratory tester as a core part of a given team. They take full responsibility on deeply understanding the domain, and often become a real expert that can identify domain level misunderstandings and poor mappings on to the solution domain. These issues are amongst the most expensive to fix, and the payback for this level of investment is ridiculously huge.
So I wish we had a more distinct term that would help distinguish this older approach of QA (or SQA) with the dysfunctional patterns we've identified from the newer exploratory testing role. Something like "subject matter validator" (naming is hard).
QA tends to make iterations slower. At the very least, there is some added step with added handoff. That's assuming there's not a lot of back-and-forth between QA and dev. Worse, they often add special test environments (with potential problems) and test passes that take a long time, so even slower still.
There's also slowdown due to the Brooksian "Million Man Month" problem. If you have a team that's having problems hitting a schedule, adding team members isn't going to fix it.
And there's the fact that the only bugs that matter are the ones customers see. More so, the bugs that impact customer use. Customers always seem to find those bugs no matter how much QA you do. And QA gumming up triage processes and filling backlogs with real "bugs" that no one cares about doesn't help anything.
I could go on but this is long-winded enough.
IMO, the way QA has been done in the past completely justified the move to eliminate QA a decade or so ago. That doesn't mean there can't be value in it. It does mean that it can't gatekeep or even slow down progress. QA needs to be committed to shipping product and trying to prevent customer-impacting bugs.
It was a very strawman argument, as if we could only have a 90's style split where QA only did testing, or you could have modern developer testing and no QA.
I think modern dev testing + exploratory testing is the best I've experienced, but it has been challenging to explain that. Software organizations are imperfect places, and sometimes it's hard to push against popular trends.