The Actual Cost of Tests – Part II


You think I was finished? I haven’t even started. The first group of associated costs to tests were around the fact that tests are, in fact, code and as such are susceptible to all kinds of flaws that come with them being code.

But that’s not all folks! There are more hidden costs that you pay for in opportunity cost: what you could have done instead. Yet you spend time and money on the following:

Longer test run times

The more tests you add to your test suite, you’re adding time to its completion, and therefore lengthening the feedback cycle. It may not mean much for a test or two, but it becomes apparent with hundreds and thousands of them. Once you got addicted to feedback (this is the good kind of addiction, mind you), you learn to wait for it before moving to the next thing. As the suite run time gets longer, and the feedback cycle makes you wait, you’ll either decide that it’s not worth waiting, or continue to wait, instead of moving to the next task. There are many things you can do in order to make a suite run faster, but if you don’t do them, the cost is there, accumulating.

Code lock-down

However well you write your tests and code, you cannot eliminate coupling between them. It is true, interfaces and abstractions are better than direct coupling, but they are just a lower level of decoupling. And coupling leads to anger, which leads to hate, and finally – suffering.

I mean lock-down.

The time will come when you will need to change the code’s interface. At that point, you’ll have to choose between changing that code, and its tests, or leaving it as-is and not pay that price.

It’s like you’re stuck in a middle seat on an airplane, wanting to go to the bathroom, but thinking whether to bother the aisle seat dweller. Sometime you don’t. And then you get angry, etc.

People tend to leave code as-is for a reason: It’s risky and costs too much. When tools do that for us, it’s a no-brainer. But when we need to do this manually, the coupling we introduce may encourage us to leave the code like that, rather than change it for the better.

Then there’s maintenance

We can define test maintenance caused by tests as the work we do on the tests, but doesn’t add any value. Remember that lock-down? When we need to change an interface, the tests don’t compile. So we fix the tests that passed a minute ago.

While this definition doesn’t cover actual functionality changes (when we actually need to change the tests for testing that functionality), we think of this work “maintenance” too. Thus, anytime we touch our tests, we’re counting that as “non-productive” work, and we want to abolish it. The fear and loathing of test maintenance is a big cause of people dropping testing.

Maintenance doesn’t just include the re-writing tests. Managing builds, deployment for testing, and the test suites themselves takes a lot of work, which the team sometimes outsources to IT or DevOps or whatever they are called in the organization. This is work we don’t see as productive, and it comes directly from having tests.

Those bugs aren’t real

We don’t just pay in writing tests. When a test fails, we rush to see what happened. And we may find that many times, it wasn’t a bug that the test found. It failed because the tests themselves were buggy. Or that they were not isolated and independent from each other.

Or because we’re not good at writing tests. (I can help here, by the way).

Bad written tests costs a lot of time. When we’re starting an implementation process, we focus on the effort of writing, but we don’t see how bad tests will haunt us for a long time. Sometimes a lot more work is wasted on bad tests than writing them.

How much for that test?

There are visible costs, deferred costs and even imaginary costs. But we pay for those costly tests in blood, sweat and extra hours.

Leave a Reply

Your email address will not be published. Required fields are marked *