Comment by ChrisMarshallNY | Hacker News Reader

Comment by ChrisMarshallNY | Hacker News Reader

ChrisMarshallNY

One of the "philosophers' stone" goals of the software industry is to completely replace human testing with automated testing.

Basically, I think automated testing is a very good thing, and we should definitely try to do as much of it as possible.

So we can clear the way for more useful and meaningful human testing.

I've always thought that the engineers in QC should be just as skilled and qualified as the ones building the product.

Part of what they should do, is design and build really cool automated tests, but I think that they should also be figuring out how to "monkey-test" the products, and get true users (this is 1000% required for usability and accessibility testing) banging on the product.

"True users" != anyone even remotely connected with the product development or testing, beyond a contract agreement.

But I'm kind of a curmudgeonly guy, and my opinions are not always welcomed.

I do write about why I prefer using test harnesses, as opposed to [automated] unit tests, here: https://medium.com/chrismarshallny/testing-harness-vs-unit-4...

> One of the "philosophers' stone" goals of the software industry is to completely replace human testing with automated testing.

Generally the people who say that don't have a clue - I meet them monthly: "We can save X money by writing a Selenium script and firing the whole QA team!" is typical.

I'm managed QA teams at a few companies in telco, boxed software and SaaS.

And people invariably get all huffy when I say, "Are you going to assign one of your programmers to review changed screens before integration and deploy?"

Then suddenly their eyes glaze over since they just lost a partial headcount and accepted even more responsibility for the release.

What I want are world-class manual testers that understand all of the product features, know how to approach testing it, what the problem areas are, and can communicate the issues to developers.

Luckily I know a few, and I wouldn't call what they do "monkey-testing" at all - their opinion of a release's quality is the only one that actually counts to me.

What I don't want are programmers writing Selenium scripts with no understanding of the product, then moving on to some other project and abandoning half-done tests. That's what you get when you fire your QA team.

> I've always thought that the engineers in QC should be just as skilled and qualified as the ones building the product.

World-class QA people are skilled at manual testing. World-class automated test programmers are not QA, they're called programmers.

For developers, what you can do is make the software you write testable. For web applications, add id's to all buttons for example, which are usually required for automated testing software to know what to specify.

Also, if you crapped out a feature (did a copy and paste of other code with "it works for me" local testing), just tell your QA person, "I crapped it out. Please take a look." so they know they'll need extra time to look at it.

>I've always thought that the engineers in QC should be just as skilled and qualified as the ones building the product.

I agree, but then they need to be equally well payed... and I don't think it is possible to find a manager willing to pay them as much

Microsoft used to do it.

It’s probably a matter of how Important your application actually is. If it’s dealing with critical numbers or features that have a real cost when it breaks, then the investment into good testing pays for itself.

I think this is important and requires a big shift in how most organizations think of QA.

A lot of QA being done is very mechanical and and done by junior staff offshore to keep the cost as low as possible. This causes the value to be low too, for example, the QA team for a F100 company I worked with would meticulously test against specs and file bugs such as "the error message is misaligned on page XYZ". Which was true, but they missed that the error message didn't make any sense at all.

Improved automated testing has the opportunity to free up people to move from Quality Assurance (preventing defects) to being the voice of the user and ensuring quality products. This shift needs a completely different skill set and mindset, but equally it needs organizations to rethink both the cost of software engineering and the value it creates.

I worked for a Japanese company that was renowned for Quality (with a capital "Q").

In the US, having "Quality" in your title often means that you are in a dead-end job.

In that company, it meant that you were an elite, and that you had considerable power over Engineering.

It also meant that you were a completely anal-retentive S.O.B.

They had spreadsheets with 3,000 rows (each row was a test -usually "monkey" test).

If even one of those rows got a red "X," the whole shooting match would come to a screeching halt, with heads rolling around like foozballs.

It also meant that they double-checked every bug report six ways to Sunday. If they reported a bug, It. Was. A. Bug. No ifs, ands, or buts. If you questioned it, they would get quite huffy; which was not a good thing (see "power," above).

Management kept the QC organization quite separate from Engineering, and they often had an adversarial relationship; which was sometimes encouraged.

This led to engineering departments having some very large testing teams; often outnumbering the engineers. The engineering departments would be penalized for bugs found by the official QC organization, so having large in-house testing teams was worth it.

Their QC doesn't work especially well for software. They would get frowny faces, when I'd suggest automated testing, or process quality best practices.

Quality was always treated separately from construction. I could never quite agree with that, but it also meant that I was "on my own," if I wanted to try using modern quality engineering techniques.

Their [hardware] products are damn good, though. What many companies would consider minor quality issues are treated like Extinction-Level Events, at that company. They've been doing it for 100 years, so it's difficult to argue with them.

> The engineering departments would be penalized for bugs found by the official QC organization, so having large in-house testing teams was worth it.

Some more details about that behavior are:

In Japan, employees start with a zero score, then for each mistake lose a point. So they will analyze/block anything that could potentially cause a demerit. Obviously that's the polar opposite of "move fast and break things."

Regarding the "3,000 line spreadsheet", yup, that's normal there and part of why their meetings often run to midnight. Gotta check and double-check every row before moving on to the next one. :)

If you want to learn more about these cultural differences, read patio11's excellent posts about working in Japan.

> their meetings often run to midnight.

I still have PTSD from that.

Nothing quite like a 14-hour flight, massive jet lag, and an all-day (and all-night) meeting in a packed, cramped, overheated conference room.

Totally unrelated, but your anecdote reminds me of Japan's WW2-era history - the animosity between the Imperial Army and Navy was so bad that the army built their own submarines.

I study WW2. To expand on that:

- Correct. Before the Doolittle Raid, they weren't cooperating with each other. Things totally changed after Tokyo was bombed.

- One of the reasons for the Pearl Harbor bombing was because the Japanese navy had nothing to do, while the army was engaged in China. But had the navy and army been coordinated, they could have occupied Hawaii.

- before Guadalcanal, the US Navy and Marines weren't very coordinated. After the Marines were abandoned there without half of their equipment and supplies, a protocol was developed that improved communications.

Until today, the Air Force and Army fight over who can operate aircraft on behalf of the Army.

And for specialized products, you want a range of testers from "this is my first day working with this" up through "I send you bug reports so often your helpdesk knows my name."

Reply on news.ycombinator.com