Functional Tests As A Tree Of Continuations (Erlang)

One good thing about killing the state each time is that... you kill the state. I'd be worried about testing an external service through http in this style. Is it completely stateless? What happens with the session? Can I test "cancel" on the same step if I already did "continue" on it? What if the order is different? What information do I have to rollback after trying each branch? etc.

I know he addresses it in the database example (<<For modern web applications, all the state is stored in a database. Therefore to make a "copy of the universe," we just need a way to make a copy of the database to pass down to the child tests>>), but then he follows with testing over http, where he doesn't save the state in any way. It's just a bit confusing this way.

If you do testing in the wrecking ball approach, at least those things are in plain view. Just kill the state every time. If it doesn't take more than a couple of min. to run all the tests, everything's all right.

Cool idea.

In my job we are working on business process engine, and we have implemented many long business processes that involves user interactions and calling plSQL code.

Testing these processes manually is pain, we have tests using soapui, but writing tests in linear fashion is repetative and tedious, our test suite is lagging us. Also we use big databases and our tests unfortunately aren't isolated - we have to prepare data manually for tests to run.

This tree-like approach looks great. One problem is - the only way I see to create "stack of databases" is to use savepoint - rollback to savepoint around every node in tests tree.

We use hibernate, so rollback to savepoint won't be enough - we'll need to disable second level cache, and to do sth about first level cache - maybe flushing after each rollback to savepoint will do.

Anybody has experience with hibernate and rolling back to savepoint?

This reminds me a lot of "context blocks" as implemented in the Ruby testing libraries Shoulda and RSpec (and probably elsewhere). A context is a named code block which can contain a number of tests, along with some common setup and teardown that's run for each test. Contexts can contain contexts, and tests in nested contexts get all the setup and teardown from their own context and its parent contexts, in the order you'd expect.

I've used nested contexts to achieve much the same sort of thing as Evan talks about here: high-coverage functional testing of a sequence of actions. They have the same advantage that the length of the test stays linear with the number of actions, although unlike Evan's callback-style tests, they didn't stop the test tree as soon as a step failed.

I realize the bad example wasn't actually doing anything and the tree example does, but the tree looks like it takes a great deal more effort to actually use.

I have a better idea: Erlang already handles green threads extremely well, how about providing a "test mode" switch in the VM. When a branch is encountered in execution the VM spawns new threads as needed and just takes them all. If spawning a new thread would put us over some thread count threshold then just put it on some queue for when there is space for a new thread. Then you can just put pre and post conditions on all your functions and you'll find out if any possible branch traversal causes one of these conditions to fail.