Is Test Automation Worth It?

Published by Gil Zilberfeld on

test automation Maintenance Cost

So here’s a silly question: Why do we need test automation?
I’m going somewhere with this, work with me.

Testing and test automation are tools to reach a goal. The goal is to have a picture of quality of our system, or our product. There’s a lot going into this picture, including the level of testing, testing the important stuff, the right stuff, over time.

A lot of things.

Test automation gives us a better picture. It’s a broader picture, or deeper resolution, depending on where you want to take the metaphor. Whatever image size we can reach testing “manually”, with test automation, we can get more image information.

Now, here are a couple of things to ponder. First, does your testing effort give you a good picture? Second, does automation give you a better picture?

If the answer is yes to both, then you can ask: Is my automation effort worth it? Including the writing, the maintenance, the whole shebang.

Is all the work worth it?

Because, if you don’t get that value, you better change direction. If you’re hell bent on more tests, but they are flaky on end up fixing them all the time, it may not be worth it. 

Maybe you should stop digging.

I’m not saying drop automation. But first, know your goal. Then, see if the road leads you there. And then see if you got the coin to get there.

And speaking of coins, if you want to get some coins in the webinar box, check out the next webinar on “10 Expert Postman Testing Tips”. It’s going to be wild.


5 Comments

David V. Corbin · October 18, 2022 at 1:07 pm

First and foremost, I agree with “Stop Digging” (unless your actual goal is something involving the preparation of a hole)…

But mainly I DESPISE repetition. of manual effort. I a human does something “definitive” (i.e. it can be defined as “do this expect that”) more than once I see that as the worst kind of waste…

So, thinks like “flaky tests”…. Yup, a REAL problem…. but experience has shown that it is almost invariably tied to other “flakiness”, perhaps in the code base, perhaps in the work planning, perhaps in…

So, dont stop testing because your tests are flaky, get to the root of the issue as to why you cannot have a solid stable platform.

Gil Zilberfeld · October 18, 2022 at 2:52 pm

Thanks David! I’m all for valuable automation. Unless it isn’t. It’s not about stopping testing, but if you don’t know what you’re getting from it, and how much it costs you, there are other ways. Relasing something and getting real customer feedback is also testing. I wouldn’t recommend it, but hey it maybe better than what some folks have.

George Dinwiddie · January 10, 2023 at 5:56 pm

I like test automation to demonstrate that the system under test does what it is supposed to do, and doesn’t do what it’s not supposed to do. Yes, exploratory testing for the unanticipated, and user testing for suitability are also important. For me, test automation typically has less involvement in those.

I’ve also seen test automation become a maintenance nightmare, and have identified two key components to this happening.
1. The tests focus on implementation and don’t describe the intent.
2. The tests aren’t treated as first-class code and written in a maintainable fashion.

That’s why I wrote Evolutionary Anatomy of Test Automation Code (https://leanpub.com/EvolutionaryAnatomy/). This inexpensive book walks through the process of developing an application driven by tests, and offers strategies and tips for doing so in a maintainable fashion.

    Gil Zilberfeld · January 13, 2023 at 4:38 pm

    Thamnks for the comment George!
    And for everyone who thinks this is spamming, please go read George’s book. He know what he’s talking about.

    David V. Corbin · January 14, 2023 at 2:09 pm

    @George – Yup, I use the term “definitive testing” : those things where yyou can document a series of steps and an expected outcome. THose are great to automate. They there are “subjective” or “variant” where a set of steps and outcomes can not be predetermined (“Which is a better shade of blue”?, “Which Invoice form is better to use, why?”. On a fairly often basis the execution of these tests (Which MUST be human driven until AI is MUCH better, thus not automated) iis the creation of new “definitive tests”.

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *