Comment by jrockway | Hacker News Reader

Comment by jrockway | Hacker News Reader

I sit somewhere in the middle. Unit testing GUI code is going to slow you down. Unit testing your core algorithms is essential. That is the stuff that must never break, and where breakages would not be immediately detectable. The unit tests ensure that you are always sure that that code works.

(Real world example; testing somewhat hairy parsing code: http://github.com/jrockway/moosex-runnable/blob/master/t/arg... I test every case I could come up with, so I could be sure that it works. Better to spend the time typing this file than to be hit with a weird misparse on the command-line, where you are probably not in the mood to fix the parser.)

On top of the unit tests, I do integration tests, to make sure sections of my app mostly work together. This is somewhere below "Request -> Response" and somewhere above the individual routines/classes. In the web app case, this is mostly "instantiate an object graph; call the methods that provide data to the rendering functions; make sure the results make sense". (I assume my template renderer and object database work; those have their own test suite, after all.) I also test stuff like typical sequences of actions that share some state (usually "the session"). I neglected that once and it caused me lots of problems.

But anyway, GUI testing == waste of time. As long as the buttons you click generate the right events, the GUI works. This is the sort of app Zawinski was working on, and this colors his thoughts on testing accordingly. Your mileage may vary.

Why exactly is unit testing of GUI components a waste of time?

Because it takes a lot of time and effort but does not provide much benefit.

OK, but why?

Is it not useful to know that various operations and navigations through the GUI still work as expected?

And if it's a matter of a poor benefit/effort ratio, why are GUIs so hard to test?

Tools like Swinger are pretty easy to get rolling with for testing Swing UIs. Where do these things break down?

Are these things worth fixing?

They break down because UI testing tends to rely on Strings: either labels for controls or embedded ids of them. It's really, really easy to have those change on you, and when they do the tests become difficult to debug: if the button with the id "Search" isn't found, is it because something broke to cause the button not to show up, because the id has changed to "SearchUsers," or what? And if I need to remove the "Search" button and replace it with somethine else, how do I figure out which tests to change? The high-level nature of the tests inherently makes them much harder to debug, because the test could have broken for any of a hundred reasons.

In other words, with unit tests the linkage between the code being tested and the test tends to be pretty tight; with UI tests it tends to be very, very loose, and that makes the tests correspondingly more fragile and much harder to debug.

We essentially do typesafe metaprogramming on our web UI that generates compile-time-checked constants for all the labels, buttons, etc. so that our tests don't compile if the UI changes, which has gone a long way to keeping the tests stable; it's the best solution we've come up with, but it's a huge investment, and our attempts to do testing of our Swing client have met with less success so far.

In my experience, I have sunk a lot of time making sure that foo div has bar css class when the quux link is clicked... but it has never saved me much time. I have to click through the site regularly (content changes, "does this work in IE", etc.) anyway, and errors are usually noticeable immediately.

Other problems I've noticed are that they are either so specific that they fail for no reason (it was supposed to be red, not green!) or too general that they don't catch issues that would annoy users.

If I could have these tests for free, I'd take 'em. But since they're expensive and don't get me much, I don't bother.

(BTW, if you have complicated algorithms in your JavaScript; refactor so you can test them with Rhino on the command-line. Don't do this stuff in the browser!)

Anyway, I would be interested in hearing your UI testing success stories.

"Anyway, I would be interested in hearing your UI testing success stories."

For Web apps, I use Selenium. I use the Firefox plug-in to record scenarios,with some hand-tweaking to replace any odd xpath stuff with more robust references to IDs. I then periodically run suites of UI tests to see that things still behavior as they should.

Yes, there are times when page content changes and breaks a test, but it's trivial to see where that is happening, so I've not had a problem keeping them up-to-date.

And having automated integration tests is way faster and more reliable than manually clicking through a site. I've caught numerous bugs this way, mostly in pages that do not typically get much use in real-life (but tend to be the first thing a clients tries when showing off code. Go figure.)

So, with Web+Selenium there's not much overhead to creating and maintaining a set of tests. It's a big win to be able to kick off a full suite and automatically run through a site far faster and more reliably than I could by hand.

It's not so good for desktop apps. I've been looking at Swinger, which marries Cucumber with Jemmy, for integration testing of Swing apps. As far as I know there are no good tools for recording user actions, so tests need to be constructed by hand. This makes it harder to assemble tests that capture assorted complex interactions.

But, at some point, someone has to actually test the app itself, so while it's time consuming to assemble automated UI tests, it may pay for itself over time since it reduces the effort in manually walking through the app.

So far neither of these would replace unit and functional testing, and neither catch all bugs as experienced by the end user, but they do reduce the number of problems that make it into a release. For Web apps, the effort spent is well worth it. For desktop apps, I've yet to reach that balance. However, things keep improving.

Often, there is benefit, but it's quickly swamped by the maintenance effort.

In the group I'm working with now, we're transitioning to all interactions with the user being abstracted into an AppModel domain object, which will describe user interaction in an abstract, non GUI or other interface-specific way. We will be implementing meta-level and meta-syntax-driven automated coding standards to detect probably violations. In addition, our (now incomplete) Unit Test suite will be run against a Code Coverage tool against the change requests. So any new change requests will have to have coverage in a Unit Test, which will be detected by a nightly script. (The above tools are based on the Smalltalk Refactoring Browser parser & libraries.)

One benefit -- we will be able to test everything in Unit Tests and leave out GUI specifics!

Another benefit of the AppModel architecture, we can eventually publish our abstracted applications as Web Services, and get out of the business of maintaining nitty-gritty GUI code. This will make that group 10X more valuable to their corporation, and they will eventually be doing only half the work!

I've found that it's often worth unit-testing functionality that depends upon complex combinations of GUI state. For example, "Show the menu only when there is a selected bar and the baz list contains at least two items that can be foozled together." These interactions are very often wrong, and even if you code them right, it's nice to have a record of what the spec was so when someone comes along and says "I can make this code much cleaner if I eliminate the check for foozling" (or worse, they change the definition of foozling), the test breaks and you remember there was a reason you put it there in the first place.

I find that it's a waste to test things like appearance, position, labels - basically anything that should be in CSS or other declarative specifications. But UIs often have quite a bit of actual logic in them (ironically, because real humans are often strikingly illogical), and that should all get tested.

Maybe a better way to put it is that testing GUI components is not a complete waste of time, but it gives you a lot less bang for the buck.

Reply on news.ycombinator.com