We tend to categorize different types of tests according to what they cover. Unit tests cover small portions of code, usually a method or a class, while we mock the rest of their interaction. Integration tests cover several components in concert, and then mock the other boundaries. System tests and their bigger brothers, End-to-End tests cover more and more.
Those categories are semantically correct, and they are relevant in the context of “how many should we have”. As in the testing pyramid:
The testing pyramid tells us that we should have more unit tests than integration tests, and more integration tests than end-to-end tests. There are a few reasons for that:
- We can cover more code with smaller tests. Mathematically, covering small parts of the code is of linear order, while longer workflows (that covers different options of interactions between components) becomes an exponential order.
- Unit tests are easier to write, because they require less setup. System tests require larger set-ups that are harder to write. While we can write both, the chance is that we’ll write more unit tests than system tests.
- Unit tests are less fragile than their larger counterparts. System and integration tests usually depend on environments and available resources, and if they disappear for some reason, the tests fail, although that is not what they are actually testing. i
- There are many paths in the code that can’t be covered by big tests, and can be covered by unit tests. You may question why this is so (I usually do), and usually the answer is YAGNI. But because there’s code, it better be tested, and unit tests do better job at that.
- When using Test First, TDD’s feedback cycle is much shorter than ATDD’s. TDD lends itself to producing more tests, and those come at the unit level.
So you’re probably asking – if unit tests do such marvelous job, why do I need the other tests?
The answer, my friends, lies in the upside-down test-trust pyramid:
Because unit tests don’t test the system as a whole, and tested system flows prove that the system works, we tend to trust them more. The longer the flow, the more trust it gains and our confidence in the tests grow.
Oh. So end-to-end tests must be the best of the batch. Why not write more than those?
We can revisit the former list, but instead, I want to talk about value.
What is the value of a test?
Automated tests (and tests in general) give us feedback. That’s their main value, but not the only one.
You see, if every test passed, the value of the end-to-end test would be the largest. It gives us more confidence that our system works, compared to unit tests that cover the same code.
But that changes if it fails.
When we have a system test failing, we don’t know what to even look for: Is the problem functional? environmental? At which point of the flow did the problem occur? However, when a unit test, covering some of the code that the same failing system test goes through, we’ve got a more focused, isolated description of the problem, which is easier to solve.
So what kind of tests should we write?
Going back to the test pyramid, more unit tests than system tests. We’d like the confidence the green system tests give us, but we’d like to solve the problems with their smaller siblings.
Remember: Green system tests give us more confidence. But the value of a red unit test is bigger than a red system test.
0 Comments