Comment by marvin | Hacker News Reader

Comment by marvin | Hacker News Reader

Parent Full thread

Hey, I have a question for all you TDD fans. In my (still short) programming career, I have only stumbled across stuations where automatic testing looks impossible to do in a sensible way - deploying a patchwork of code against huge platforms like SharePoint or working with APIs like COM that don't lend themselves very well to testing, and where the code is "interface" heavy rather than "logic" heavy. Recently I've been looking into iOS development.

I also get the impression that most IT work today mainly involves working with huge libraries and APIs, and that automatic testing therefore is hard to implement in a sensible manner. I very much get the idea of automatic tests (and I've wished for them quite a few times, where they've been very hard to implement because the API seems to get in the way). But are there really so many applications for them unless you are building a huge, monolithic application where everything is defined in advance? Seems to me, like a lot of people have pointed out, that it will slow you down when prototyping.

This is not a criticism of TDD/automatic testing, but I just don't see how to create good tests when most of your code is just glue between different libraries and most of your time is spent reading documentation and chasing bugs in your library. Would be really cool if someone could point me to an overview of these things. Am I just in the wrong organization?

You're asking for integration tests, and yes those are hard to do, esp when the integration is against a large deployment of third-party or existing API code like you describe.

What you want to do in that case is isolate the "glue code" if you can and test its assumptions in isolation. Wrap the API dependencies in an interface and inject mock objects to play the role of that API. This is really what mocks do well.

If your code is also bootstrapped by some special plug-in hook that is hard to emulate in a test environment, like a MS SharePoint or Dynamics thing, then you should isolate the code in question from the Class that implements that hook, so that a test can boot up that code just like the plugin would. Interfaces are probably a good option on this end as well.

So, you often can't test your production code in an integration environment exhaustively, but that's OK in most cases because A) a truly exhaustive integration test is probably a combinatorial problem and not realistic anyway and 2) you'll just be proving that your third-party API works as guaranteed, which is probably not your highest risk and not worth the trouble.

Your real concern is to test the assumptions of new code and also create the TDD discipline around that code which tends to make for better code.

At Circle, we write three kinda of tests for this sort of thing:

1) "Are we using the API right?". So we have a test that logs a test user into github and reads their API info, for example. This sort of test shows that we have integrated the API correctly. If it passes, we can assume the rest of the API works, because we are using it correctly.

Obviously, we can't truly know this, so it depends a little on the quality of the API - kinda like trusting your compiler or OS. But the APIs we use - EC2 and Github, are largely bulletproof so long as the service isn't experiencing failures.

2) Stubbing out the API code and checking our logic. For example, we need to test that the code which manages how many builds we run simultaneously works, but we don't want to run builds, pull from github, etc. So we make the function calls return fake values, and test the logic.

3) Integration tests: Run the full code, with no mocks, no stubs, across an entire "process": do an entire build from webhook to UI, including starting up machines for it; or maybe selenium tests that the OAuth login works.

I visualize tests as a graph: integration tests and API tests provide thin edges between strongly connected components of unit testing.

I like to think in these scenarios, you write your tests for your code as:

   given input foo, my code calls library bar and expects a return of baz

You don't need to test the APIs or the libraries. You can mock the return and stub the original method call (if you need to test in isolation -- think APIs over the network).

Reply on news.ycombinator.com