Unit Test Attribute #9 – Deterministic

Standard

Gil Zilberfeld talks about how unit tests and integration tests being deterministic increases our trust in the unit test.This is the 9th post in the Unit Tests Attribute series that started here. To learn more about testing, contact me.

I keep hammering on trust and how it’s crucial that we trust our unit tests. If unit tests are deterministic, they raise the level of our trust.  If the unit tests are not, we may question their result, which will be followed by questioning other unit tests as well. By the way, with integration tests, the problem gets a lot bigger.

Let’s start with a simple example. What’s wrong with this picture?


This is the tested code. So we can write this unit test:


Which passes. Until a specific day arrives, and then the unit test fails.

Or this code:


And its unit test:


That passes, as long as the unit test finds the file where it expects it, or somebody edits it “the wrong way”.

These may look like dependencies we can’t trust all the time.  It’s more deep than that.

Never Assume

When we write code, we have many assumptions. The “happy path” we usually code first, when we assume nothing will go wrong. The other “negative” paths are the the paths where something might go wrong. Unfortunately, we can unit test and code only the things we think of. Actually, writing helps us think of other cases. The downside, is that if we use a library we didn’t write, we think less of the side effects.

To make sure we remove uncertainty from our unit tests, we need to remove being dependent on:

  • Geographical location
  • Time
  • Hardware speed, CPU, memory
  • Files and data, both existing and missing

The obvious solution is to mock the dependencies, or setup the system to be independent in the unit tests. By mocking the date we can check the “before” and “after” cases, or “plant” the file there and simulate reading it. That goes for integration tests in some cases

However, mocking comes with its own drawbacks, so you should consider the trade-off.

And sometimes, mocking isn’t an option. We have had a performance test, that was supposed to run under a certain limit. It started failing and passing on and off. Instead of checking the problem we said: “Oh, leave it, this test is just like that”. And we missed opportunities for fixing real problems.

To handle the root cause – assumptions, we’ll go back to our friends. Review the unit tests and code, and try to either validate the assumptions, or invalidate them (in that case, remove the code and unit test). Unverified assumptions may cause not only bugs. The code you’ve added will make it harder to add new code in the future. Use code that not only works, but that is also valid.

Next time: Isolation.

Leave a Reply

Your email address will not be published. Required fields are marked *