The Actual Cost of Tests – Part I

Standard

We talked about what we perceive as the cost of tests: slowing us down.

That does not mean that tests are free. There are actual costs associated with tests, although they may not be what you think.

I want to start with the cost of writing them. But not because writing tests slows you down – any line of code slows you down. All code, including tests are liability. As we’ll see, almost any cost that is associated with code has a test counterpart.

Refactoring

Like everything we write, including this blog post, the first version is not so good. The principle applies to code and tests. Because it’s not so good you’ll pay, and the cost goes up linearly, and sometimes exponentially, with how crappy it is. If you copied lots of setup code, when you’ll redo it, you’ll go through that setup code and try to understand what you needed from it. If you’ve picked lousy names, you’ll need to fix them, after you found out what that test actually did. Refactoring is in an investment, so it has a cost part. The closest you do it to the coding, it will be cheaper.

Debugging

When code doesn’t work, we need to debug. Of course, that’s why we have test for, and the amount of debugging is less than debugging a full system. And yet, if your tests are obscure, rely on big setups, the code they test is complex, and has some evil singletons hiding in the bushes – you’re in for some fun debugging time.

Oh, and when they fail inconsistently? A full on debugging party.

Bugs in tests

Of course, we can’t really talk about debugging without mentioning bugs. It is possible, and surprisingly easy, to get bugs in your tests. It is also very easy to identify and solve these bugs with code reviews, but still. The cost of having bugs in your tests starts with debugging, but even more so, it introduces a giant fear factor into your life. You start asking: If there’s a bug in that test, what about the others? Can I trust these tests? And what about that moron team member of mine, whom I don’t trust at all? Should he even be allowed to write tests?

Not trusting your tests could be a tipping point on your quest for quality. It can topple the whole building.

Cross coverage

Some of the tests you write may be waste, because you have other tests covering that functionality already. That happens when you have bad factored code, or that your tests cover too much. Or you may just “want to be sure”, so you add a couple more tests. Any of these additional tests can have bugs, require debugging require refactoring. See how all of these accumulate?

This is just the opening show. There are more costs coming. Stay tuned.

Leave a Reply

Your email address will not be published. Required fields are marked *