Unit Testing Anti-Pattern: Misleading Tests

Standard
This series goes through anti-patterns when writing tests. Yes, there are and will be many. 
TDD without refactoringLogic in testsMisleading testsNot asserting
Code matchingData transformationAsserting on not nullPrefixing test names with "test"

You’re testing the right thing, and the test passes, and everything is cool.
Or is it?

Check out this test:

This test looks ok. I bet that without looking at the implementation, you can guess how the result is created. Classic black-box testing.

What about this one:

For the same implementation, method and arguments this test checks a different aspect, that may be important for us.

The problem is usually not over-testing. If I saw these two tests, I would say both are important and therefore we should have both. The problem starts when we use the familiar pattern in the first example instead of the second. When what we’re really interested in is the more specific aspect in the 2nd test.

It’s the same coverage, so what’s the problem?

Shine the flashlight on the problem

Remember when we talked about test attributes? The first, and most important one was validity – tests should check correct behavior. Both tests do that.

But then, there’s another attribute that’s important: Accuracy. Our tests should pinpoint the problem when they fail. When tests fail, it should show us a way to the problem, so we can solve it quickly.

The only way build these future flashlights, is to use our tests to specify the important aspects.

Don’t go by coverage and don’t cast a bigger net. Even if the tests check for correct behavior, and fail for the right reason – if they don’t lead you to solve the problem, they are misleading.

3 thoughts on “Unit Testing Anti-Pattern: Misleading Tests

  1. A good post. However for “unit tests” I disagree strongly with your usage of “tests should check correct behavior”. Any test that is written by the same person as the code (or even same pair) can not do this.

    The developer may “misunderstand” the user story/PBI/task. The work item may contains errors. There are many scenarios where the developer believes it is “correct behavior” but it is not.

    The best that these tests can do (and it is indeed a very important thing) is to capture the developers *intent*.

    • Gil Zilberfeld

      David,

      Tests are always written to capture intent. Either intent of implementation or behavior. The best way to uncover the wrong intent, is by review. The other way is to excessively test, using different methods and test types on the same code, which still doesn’t guarantee it will catch anything. Tests don’t guarantee anything – they raise our confidence in what we develop. Still they are not the only thing that do that, reviews help.

      • I partially agree. Two elements…

        1) There is a category of tests known as “pinning tests”. They simply characterize (often at a very deep level) the current behavior and then fail if there is *any* change. These tests to not identify “good/bad”, but rather “same/different”. If the concept I swell understood these tests are extremely valuable.

        More directly related to my previous reply, there is a fundamental difference between “self validation” and “independent validation”. The ” aim or plan” of the CODE is something that specific to programmers. Other stakeholders (including customers/users) care not about the code, but rather the overall behavior of the system.

        I have seen far too many times where the unit tests [even with reviews inside the team] and the code contains a common “misunderstanding” of the fundamental goal/need.

        I am very much a proponent of extensive unit testing, but I strongly believe that these types of tests are merely one aspect out of many that need to be addressed in order to achieve optimal results.

Leave a Reply

Your email address will not be published. Required fields are marked *