Hacker News Reader

"There is a time and place for unit tests... I prefer using integration/functional test."

Personally, I've given up on categorization entirely. It's just "automated tests". People seem to be very prone in getting caught up in the question of whether it's a "unit" or "acceptance" or "integration" test when that's a completely uninteresting question.

Even "Test what, not how" is only a rule of thumb; if I've got a complicated "what", I'll sometimes at least probe the implementation to be sure I'm on the right track during development. Of course, if I change the "what" I just nuke them. I say this having done this entire cycle over the last week. My "what" was too slow and I need a new "what", but unit testing the "what" helped me get it to the point where I could benchmark it much faster than if I had tried to "not test the what". (Sometimes, you can also nuke the "what" tests before putting them into long-term storage.)

If you can avoid getting caught up in attaching goodness and badness to the terms, unit/acceptance/integration can still be useful communication terms, but I resist using them for anything else, and I don't personally use them at all.

(Also, to be clear, I consider this amplification or reflection upon your post, not disagreement.)

Categorization exists so we can communicate. Labelling abstractions enables us to grok more in our head, and communicate faster. Your second paragraph demonstrates this:

"My "what" was too slow and I need a new "what", but unit testing the "what" helped..."

If only there were higher levels of abstractions to say what you mean.

There are higher labels of abstraction to say what I mean. In fact I could describe it in great detail. My boss may not particularly appreciate me using that level of detail. For that matter you probably wouldn't either, it isn't that interesting to the core point here.

My explicit point in this case was about how the labels are actually bad, get in your way, and shouldn't be used because they hurt more than they help.

Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly for you to run it often enough to validate your design. It ends up being something you run when you think you've completed a feature to make sure it works, but not after you've extracted a class to make your code cleaner.

I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.

"Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly..."

I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well!

Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem.

Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level.

The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.

The problem with creating unit tests before the code is that it simply doesn't work for exploratory programming. I often don't know exactly what I intend to do or what the tools are capable of until I'm fairly far into coding the features. Writing unit tests at that early stage would be a complete waste of time.

If you know ahead of time exactly what the inputs and outputs should be then sure, writing unit tests first is a good practice.

I don't understand what you are saying. How can you start coding if you don't even know what you intend to do or what your expected inputs and outputs are? What are you exploring, and how do you know when you've found what you're looking for?

As an example of exploratory development that's very difficult to do test-first, I've been doing a lot of image processing work lately with the objective of improving the legibility of photographs containing text. Trying to find a way to automatically test the output before putting together a solution is a unique problem unto itself, since any given set of input parameters could potentially "solve" the problem for some images but not others.

Sometimes you have to explore the problem domain and available tools for a while before you figure out where you want to go and how to get there. It's sort of like an iterative prototyping methodology.

http://paulgraham.com/arc0.html

I don't use Arc (or any kind of Lisp) but I understand what Paul Graham was trying to achieve.

Another simple example from my work of the moment: my goal is to import points from a file (that I'm assured does have points) using a 3rd party file import library. I don't know in advance how the 3rd party library imports points, so the best test I could possibly write ahead of time is a fairly high-level integration test, "Did we manage to import a point?"

All my actual programming work here consists of adding printfs at strategic points to tease out exactly where I should be looking for the point data. Unit tests are completely useless for this process.

Come to think of it, even once I've got it figured out, it's very hard to see how to usefully use unit tests here...

Kent Beck talks about this in episode 1 of http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...

The example he works with is writing a driver for Tokyo Tyrant in java. He begins by writing a high level test of what he eventually wants to be able to do. Instead of implementing it though he comments it out and then starts to play around with potential implementation code inside the body of the test. Once he's finished playing he then gradually refactors code from the test body into the implementation until he's got everything he needs and uncomments the test.

It may well be the case that by the time you've finished exploring that you've realised the test you've written isn't very relevant. This is fine, the purpose of the writing the initial test is to guide your exploration in a productive direction. Once you have some implementation code you can always go back and reframe the test according to your new understanding.

I've had problems relying heavily on functional/integration tests.

The first problem is data consistency. Both require consistent data in whichever environment they are running. Not an insurmountable goal, but one that often leads to a lot of false failures. These lead to a lack of trust in the test feedback.

The second problem, as others have already pointed out, is that integration tests and functional tests don't tell you what is failing or why. Can I create a user? Nope. Why? 500 error. Time to fire up the debugger...

The third problem is that functional tests do not scale. Each functional test will take orders of magnitude longer to run than integration tests, which take orders of magnitude longer than unit tests. You very quickly end up in the situation where you have more hardware than people or you no longer get fast feedback.

Personally, feedback is the primary driver behind test automation. It is nice to know what I wrote will probably work and it certainly helps simplify the code. However, the biggest pay off comes from the fast and repeatable verification that what I changed has not impacted the code base negatively.