Hacker News Reader

Writing good automated tests doesn't necessarily rely on a great developer, it all depends on how you approach testing. For example, let's say you want to use Selenium to write web UI tests. One common approach is to have developers create a page object model, which testers can then use to write readable and robust tests. Creating a page object model is a simple task, and working with that page object model is a simple task, as the model guidelines define an easy to implement and easy to follow control structure (essentially all page object methods must return another page object, which means you can chain them together).

Oh and MS Paint would be easy to create a good test suite for. For what it's worth I'm a software tester by trade, so perhaps it's straightforward for someone who creates tests for a living to know how to approach it, I guess someone who was inexperienced wouldn't necessarily know how to approach it.

Do you have any suggestions on where to learn more about testing? I've gone through numerous tutorials and read through The Art of Unit Testing but I still feel like I'm writing tests just to write tests. It's not really clicking for me

Sure. First of all, even though you're clearly a developer (as you mentioned writing unit tests), I'd recommend starting at a higher level of abstraction with BDD tools, basically anything that implements the Gherkin language (I can't tell you which BDD tool to use as I don't know what language you're coding in, if you can tell me I can give you a more specific recommendation). The idea behind this is that you're writing your code to meet a specification that your client can also work with, so you can be sure the specification you're coding against is what the client wants. As an added bonus, the Gherkin scripts form a type of living documentation, providing clear information about what the application does whilst also remaining up to date (so long as all tests must pass before a new version is sent out).

Second recommendation is to look into how to avoid test-induced damage, which is where code gets bloated and more complicated in order to make it more testable. One major source of problems in this area is the need to create mock objects. I'd recommend this video from Mark Seemann as a good starting point in this area, as it looks at how you can create unit tests without mocks:

https://www.infoq.com/presentations/mock-fsharp-tdd

If you can let me know which language you're mostly coding in I'll see if I can give you some more recommendations.

I'm doing 90%+ of my code in C#/vanilla JS at the moment. I was using moq to mock out my C# objects. Thanks for the help, by the way

You're welcome. As you're familiar with C#, I'd recommend SpecFlow.

http://www.specflow.org/

As you're also using JS I'm guessing you're creating web apps, so I can recommend Selenium if you want to automate front end tests. You can code Selenium tests with C# too, and you can also abstract away the details of the Selenium implementation to use SpecFlow to write the tests. If you have access to Pluralsight (if you have an MSDN licence check to see if it's bundled with your MSDN licence, I think I got a 45 course Pluralsight trial with my MSDN Enterprise licence), there's some good courses on Selenium and SpecFlow, including one that takes you through combining the two.

http://www.seleniumhq.org/

https://www.pluralsight.com/

You've got it right on the web apps. And thanks again, this should be a big help

You think it would be easy, but UI tests automated by things like Selenium are hard coded sequences. As far as I remember, they do not run randomised collections of features with randomised data input.

Plus they do not verify application state really, just that it does not crash. What if it just looks and acts funny?

I've used Selenium. Perhaps you missed that I said I'm a software tester. This sort of stuff is my bread and butter. In terms of values, there are well established guidelines you follow when picking those values, for example boundary cases.

As for "looks and acts funny", that's why you still have exploratory testing. Automated testing can drastically cut down on overall testing time, but there's still the need to perform exploratory testing to look for quirky issues.

Developers should be writing their own tests. If a developer thinks writing tests for his own code is a waste of time that's a red flag for me.

Developers should be doing a lot of things. In my experience the trouble is that most developers are too optimistic and tend to neglect negative test cases. So most teams still need specialist testers to ensure acceptable quality.

Is this not a case for developer education? If developers are neglecting corner cases that's a real issue that needs to be addressed directly rather than having it be the job of someone else to clean up after them.

Fish or cut bait? Obviously I train developers on my teams to be better testers but that takes time, meanwhile we have to actually get working releases out. The testers work directly with the developers, they aren't cleaning up after them.

(And please no one tell me that I should only hire developers who are already also expert testers. I have to operate in the real world.)

Yeah, it's too hard to hire good programmers and too expensive as well. If you have the capability to teach programmers, then that's a real competitive advantage.

It's not hard to hire good programmers. There are plenty of good programmers available at the market clearing rate. However there are never any perfect programmers available anywhere at any price. Everyone has some flaws and knowledge gaps. So managers have to keep their expectations realistic.

That's why you need code reviews as well as testing. Another set of eyeballs can spot the soft spots in the testing regimen.

This is also a huge win for pair programming - code review is built in.

Careful to qualify that. Sure developers should write unit tests for their code when possible. But they should not be the ones coming up with the acceptance tests (product owner) and doing all the testing of their work (QA).

I've seen that argument several times "Why can't the developers test their work?" For the same reason professional writers have proofreaders, editors, translators, etc. Because it would be amateur not to have more than one pair of eyes and one brain looking at something. No matter how good those eyes/brain are, they'll miss things that would be obvious to someone else.

I'm not sure what would be so special or difficult about writing unit tests for Ms Paint.

MS paint includes user selected faunts. That's an endless rabbit hole of complexity.

No, it's not. It's quite simple.

There are only a few types of fonts available - TrueType, PostScript, bitmap, maybe others? At any rate, you test a couple of examples of each type. Paint is a bitmap graphic generator, right? Apply letters in a test font to a bitmap, and compare the resulting bitmap to an expected value.

We don't need to test every possible font there is. We only need to test enough fonts to cover the likely failure cases and known bugs.

Speaking of bugs, suppose we find a bug with a specific font, or a specific size of font, or something like that. We write a test that exercises the bug, then fix the bug. Because it's a one-off test, it's not a tremendous burden to write. Moreover, that bug will never come back undetected again.

Validation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.

You're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.

You target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.

Users don't care if a bug is due to your code or the code in a library you depend on. It's up to you to make sure things work either way.

If you find bugs in 3rd party components whilst testing, of course you try and do something about it. The point is, at some point you have to trust that the components you're working with are relatively bug-free otherwise every application will end up testing all the features of the OS it sits upon. Do you really expect developers to run a full test suite for Windows every time they develop a painting application? Note that I'm talking about actively looking for bugs, not the bugs that are apparent from some happy path testing.

Edit: Some spacing, and grammar, and some sentence structure.

+1. Putting too much faith on the test coverage is a sign of too much faith in humans.

Some of my time goes to mobile development/android these days, and here are facts:

1. On android, there is occasionally this one vendor that crashes randomly for some piece of code. Because they modified the source code for android when they shouldn't. And it's just one model, among thousands of others. And it's different model for different code pieces.

2. Android API is not exactly one would assume. They return nulls for cases when they should return empty lists, and vice versa. They do this occasionally. So you don't really know when an api returns null vs empty list, and some apis (like get running processes) should never return empty, except that it does in some bizarre cases. So your code would need to test both empty list and null, which adds to your test scenarios. Aside from null/empty list, there are many other examples.

3. When you release, all your tests work fine, at least for what they test. There is this one client that uses things a bit differently than others. You also fail 10% of the time for that client because of how they use an API. Especially in Android, how they use an API does matter. A simple example is using Application Context vs Activity Context vs Custom Context implementations. You'd assume they should all work fine, because Context is an abstraction and Liskov substitution principle should apply, but it doesn't because it's leaky abstraction. Thus, somethings works with some context while others work with other contexts, and you don't really know which.

I came to the conclusion that for most cases, testing is a must to verify correctness of known scenarios, and known regressions. However, for automated upgrades/updates, testing will not work. Not at all. Doing automated updates/upgrades, it should be a controlled release. it makes more sense to make the release A/B test way, if possible, where small population (say 1% of the users) get control/previous release, and another population of the same size but disjoint gets the experiment/new release. Then you compare your baseline metrics. Could be # of http connections, number of user messages sent, number of crashes, etc.

Testing can only cover what humans can think of. Unfortunately, we don't know what we don't know. That's the main problem.

True, testing won't always catch unknown unknowns. BUT.

When the bug that slips through the testing is discovered, the first thing to do is write a test that exercises the bug and makes it repeatable. This is sometimes difficult in the case of intermittent or highly condition-dependent bugs, but simply learning how to repeat the behavior is an important learning experience.

The bug should not be fixed until there's an automatic test that induces the bug. That way, you can test to know that your fix actually works.

That's a great ideal. But often the code has to be substantially rearchitected to allow for automated testing like this (especially for those difficult bugs, as you mention), and when the fix only requires changing a couple lines of code, it's not worth the risk.

Although it would be great if we went through the exercise of "how would we write an automated test for this?" at a high level, even if it doesn't get implemented. That might get people thinking about how to architect their code so that it's more testable from the get-go.

If the code is written with a proper testing suite from the start, it's not hard. If you're dealing with a legacy code base that doesn't have a testing harness (I've done this a lot), then the first thing you should be doing before making changes is writing tests for them!

Every time I do the "I don't need a test for this, it's just an obvious little bug" thing, and I implement a test-free fix, it fails! Every. Single. Time. Maybe it's because I'm stupid, but I like to think I'm above average. So if above-average programmers have a high risk of releasing "bug fixes" that don't actually fix correctly (or cause other problems) because they didn't write good tests, that really turns the "not worth the risk" response on its head.

It's not that it's not worth the risk to write tests. It's not worth the risk to not write tests!

That's the part I meant when i said "known regressions". If there is a bug, you write the test, it doesn't happen again... Until next refactor, because most often you'll find that the tests were written at a lower level, instead of a functional test (that tests the e2e functionality) it was written to test individual pieces of SUT such as individual methods in a class. This happens quite a lot. Devs will prefer unit tests over more complex e2e tests which takes a lot more effort to write (because they are slow, flaky, more complex). Any friction in this process will make the dev prefer the easier way to test (which does not mean future proof).

Even if you do automated testing you should still do exploratory testing, it's not an either/or situation. Automated testing allows you to cover the bulk of your system functionality, exploratory testing allows you to pick up on the unexpected quirks that crop up. Automated testing frees up the time of testers to do exploratory testing as they don't have to run through all tests manually.