When we talk about footprint, we’re really talking about isolation. Isolation is key to trust.
The “checking” part of unit tests, is really about trust. We check, because we want to make sure our system works as we anticipated. Therefore, we build a suite of unit tests that confirm our assumptions about the system. And every time we look at the unit tests result, we want to be sure a 100% these unit tests are not lying to us.
We need to trust our unit tests, because then we won’t need to recheck every time. We’ll know failure points at a real problem. And that mass of unit tests we’ve accumulated over the years was not an utter waste of our time.
We need to know that no matter:
- Where in the world the unit tests run
- When the unit tests run
- On which kind of machine the unit tests run
- Who runs the unit tests
- How many times we run them
- In what order we run the unit tests, alone or in sequence
- And any environmental conditions we run them
The result will not be affected.
Isolation means we can put a good chunk of trust in our unit tests, because we eliminate the effect of outside interference. Integration tests are susceptible to this kind of outside interference.
If we ensure total isolation, we’ll know that not only does Test XYZ has reliable results, it also doesn’t affect the results of other unit test.
There’s only one small problem with that.
We cannot ensure total isolation!
Is the memory status the same every time we run the unit tests?
Did our browser leave temporary files around the last time that might impact how full the disk is?
Did the almighty garbage collector cleared all the unused objects?
Was it the same length of time since system reboot?
We don’t know.
Usually these things don’t matter. Like in real life, we’re good at filtering out the un-risky stuff, that can have an affect, but usually doesn’t.
So we need good-enough isolation. And that means minimal controllable footprint.
- Every memory allocated by unit tests should be freed
- Every file unit tests create should be deleted.
- Every file unit tests delete should be restored.
- Every changed registry key, environment variable, log entry, etc…
I’m using Test, but I actually mean Test AND Code. So if the tested code does something that requires rollback, the test needs to do it as well.
In integration tests, these things get more and more complex.
Mister, You Are A Terrible Isolationist!
It’s not the first time I’ve been called that.
Sounds a bit extreme, isn’t it? I mean, if I test against a “dirty” database, and don’t rely on any previous state, am I doing things wrong? Do I need to start always from the same database?
Well, yes and no.
If you’ve analyzed the situation, and have written unit tests that don’t rely on previous state, that means that you’ve taken isolation into account already. So a suite of integration tests that pile data on the database and don’t clean it up, are in a context that doesn’t care about footprint.
The question is – what if the test fails? Since you’ve allowed the integration tests to mark their territory, you now have tests that are hard to reproduce. That will cost you in debugging, and maybe in even resolving the problem.
As always, it’s an ROI balance of risk analysis and mitigation. You need to be aware of the balance, when making the decision.
Next time: Truthiness.
For training and coaching on topics like this, contact me.
Image source: https://www.flickr.com/photos/scarto/3911280619/