It’s spring. Do we have clean tests?
Any automated test we write, comes with assumptions – how the code will behave, how the setup for the test will push our tests forward, and how are expectations about it are going to be met.
There’s a thing that we try not to think about, and we usually succeed. We think of our test (and the code) as the only test in the world. There’s nothing before it, after it, or in parallel to it.
Of course, that’s not true. We don’t control which test will run before it or after it, or if the CI will run it in parallel to its siblings. I mean, we can control some of it, but even if we do, that may change in a week, month or year.
Is isolated test a clean test?
Focusing on the isolated test helps us devise simpler conditions for setup and success. We disregard all kinds of unplanned interference. For example, when we assume running our tests against an “empty” database, we assume that the only thing affecting our test is what we provide in the setup.
But if the database was “dirty”, we need to think about all kind of situations. First, what kind of “dirty” are we talking about? Is it full? half-full? Does it matter?
(Yes, it does, because when we run out of storage space things start failing. When we are close enough, latency kicks in).
There’s another kind of “dirty”. What if the data sitting in the database, is the result of a failed test that ran before? And that failure left inconsistent data? Will that affect our test?
Ok, moving to another kind of dirt. Let’s say we write lots of logs during our tests – we may run into timing issue that are not apparent in production. Usually, it’s the other way around – logging slow the system down enough to hide timing issues.
I got another one for our dirt list. This time, not in the database. Running our code consumes memory (if not cleaned up over time), some of it global state that we are not aware of. Our “assumed” starting point has changed. If we were to reproduce the situation, we’d probably just run the test, and presto – problem gone!
Yeah, things are not clean as we imagine them. That leads to weird, unreproducible results. Or worse, we build confidence the system works, where in fact, it doesn’t.
How do we get clean tests?
We need to be verifying our assumptions. But, that means we need to be aware of our assumptions first.
If we assume the database is connected for our test to succeed, we should run a verification test that checks that, before running our tests. If you need enough disk space, verify that.
The other side of the coin is about clean up. If a test makes a mess (or state change, define your own mess), it can clean it up. It’s never perfect – since the code might be doing something the test is not aware of. But unless there’s a consistent mess – e.g. all logs going into the same folder – there’s only one entity that knows what the hell happened here. That’s the test.
Spring is upon us. give some caring to your tests and code. And check out the clean test workshop, it may be what you’re looking for.
0 Comments