This series deals with the implementation of a unit testing process in a team or across multiple teams in an organization. Posts in the series include:
Goals	Outcomes	Leading Indicators I	Leading Indicators II
Leading Indicators III	Leadership I	Leadership II	The plan

So far, we’ve talked about the process itself, our goals and expectations, what to look for while we’re moving forward, and now it’s time we get to the good stuff.

How does an implementation plan actually look like? A good plan includes these elements:

Training

Remember that when we start, we already have a core team, usually one, who learned the ropes all by themselves. While they can be great ambassadors or mentors, they are usually not trainers. They know what they’ve encountered, and that is usually much less than skilled practitioners and trainers, who’ve seen lots of code and tests.

The other teams, the people who start from scratch, need context, focus and the quickest ramp up in order to get started. In my introductory courses, I introduce tools as well as effective practices of testing – planning, writing, maintaining, working with legacy code, etc. In addition, I expose them to design for testability and TDD. The courses are hands on, so people can practice the different topics.

Environment preparation

Apart from having the tools available on the developer machines, we need a CI server that’s configured to run the tests and report the test run results. We’d also like to have project templates (maven archetypes, makefiles, etc.) available so people won’t need to start from scratch.

All dependencies (libraries, tools, templates, examples) should be available in a central repository. On day one we want people to start committing tests that are run and reported. We don’t want to have them bump into environmental problems and extinguish their motivation.

Coaching

This are sessions (1-2 hours each tops) when an experienced coach (either external or internal), sits with one or two people and helps them plan test cases, write tests and review tests for things they are working on. This way we transfer the knowledge of testing, as well as starting to create conventions of “this is how we test”. We focus on code that’s being worked on, focusing on making it testable and proving it.

If you start out with an external coach, it would scale up to a point. The idea is to start with a small group that can later become the mentors for new people in a viral way. The ambassadors from the pilot stage can and should support that process.

Communities of practice

We want to continuously improve the way we test, discuss and share our experiences. As we’ve already discussed, there should be forums for discussing and practicing testing. That means we need scheduled time, when people are encouraged to attend, talk about what they did, and learn from others. Test reviews, refactoring together, learning patterns – these meetings breed stronger developers.

These COP meetings are opportunities to discuss the metrics and goals, adapt if help is needed. They are engines for learning and imprvement. They also send the message from management that testing is important. As time goes by, and less coaching sessions are needed, the COP takes over as the main teaching and mentoring tool.

There you have it. In the next posts, I’ll go through a case study of deploying a unit testing plan.

 

 

 

This series deals with the implementation of a unit testing process in a team or across multiple teams in an organization. Posts in the series include:
Goals	Outcomes	Leading Indicators I	Leading Indicators II
Leading Indicators III	Leadership I	Leadership II	The plan

We talked about management attention and support, and there’s more leaders can do, in order to help us make the process work.

Remember those leading indicators? They don’t collect themselves. If we think about those indicators as a feature, there are customers waiting for them.. Management and leaders are those customers.

Turns out the people who care about the metrics are the ones who have the power to facilitate their collection, demand the reports, analyze the patterns and ask for correction plans. Who knew?

The funny thing is the collection of the metrics is a leading indicator by itself. If the process of metric collection, anlysis and feedback is done, chances are the process will go well, since somebody’s supporting it. If it doesn’t (for any combination of reasons), people see that the process is “not that important to managers”, which quickly translates to “it’s not that important to us, and me”. Even if people care about unit testing, they care more about making sure they don’t get caught on other things that managemnet does care about. Safety first.

Embracing change

Even when the process does go well, an anti-pattern may appear: Sticking to the initial plan, rather than changing course over time, which we expect leaders to do.

An example of that is around our favorite metric:  Management asserts that coverage should increase over time. Since our leaders are wise, they don’t state a minimum coverage threshold, just an indication that coverage is growing. And, we’re not covering old code, just new code. Looks innocent enough.

What happens next will surprise you (not). To keep the coverage increasing, people are encouraged to add tests to their code. But since our developers are also wise, they don’t add code that unit tests don’t make sense for (like data transformation, or auto-generated code).

So the coverage doesn’t rise. Alas, if that’s most of their code they don’t get “coverage points”, or in some measurement systems, lose some of them. Remember safety first? Let the gaming begin. The developer may add tests that are ineffective (or even harmful), just to satisfy the metric.

Overwatch

The only way to make sure this doesn’t happen is through retrospection, analysis and feedback. I’ve already said that as stakeholders, leadership can make sure these take place. But should they do all that by themselves?

Here comes the next way how leadership helps the process: creating forums for the learners to learn further from peers and grow internal experts. We call those communities of practice. Where practitioners discuss, review, analyze and get feedback to learn from the success and mistakes of others.

The chance of these forums created, and continuing to take place over time, is likely to succeed with management support. In these forums, the discussion takes two forms: the tactical level – how to write better tests, learn refactoring methods, etc. But also at the strategic level, look at the process itself, the metrics, suggest course correction, and follow up with the implementation.

Now, the more authority we give the communities, the better. We like self-organizing and self managing teams. They will need leadership to be created, exist and help when they need external resources and efforts.

Combine all these leadership support methods, and we’re on our way to a successful implementation.

This series deals with the implementation of a unit testing process in a team or across multiple teams in an organization. Posts in the series include:
Goals	Outcomes	Leading Indicators I	Leading Indicators II
Leading Indicators III	Leadership I	Leadership II	The plan

Any process we start to roll out, requires management support. If we want it to succeed, anyway.

Inside teams, if the team leader opposes the new process, she will either actively, or secretly, work against it. If she’s for it, she’ll mandate it. When the team is independent enough, and can make their own decisions, the team leader will approve of the team making their own decisions, and facilitate the team’s success.

When we’re talking about multiple teams, and cross organization processes, it’s not even a question. Not only do we need to make sure the new processes take hold, sometimes we need to make more resources available if they are not there to begin with.

Think about an organization moving from one team writing tests to multiple teams. We need to support all of them on the IT level (enough resources and environments to run the tests), at the branch management level (who works on which branch and when the changes move to the trunk), at the automation level (optimizing build performance), and coordination level (what happens when the build goes red).

To make a (very) long story short, it takes management time, attention, and a lot of pushing (nicely) to allow the process to take effect.

Oh, and there’s one more thing leaders need: Patience.

Regardless of how simple the process is (and unit testing, is definitely not simple), patience is a prerequisite. Any process implementation takes time, and we usually see the fruit of our labor down the line. Add to that the learning curve is steep, and the recipe for impatience is complete.

The learning process in unit testing seem short – if you focus on learning the tools. But until people start writing regularly, effectively, and see the benefits it takes weeks, usually a lot more.

If there are constraints and conflicts, it will take even more than that. Consider a team working inside a legacy code swamp, and a closed architecture they aren’t allowed to change. Their ability to change the code is constrained, and therefore the ability to write tests is constrained. That means less tests written, and often less effective tests at that (you test what you can, regardless of importance).

Expecting coverage to rise under these conditions, and even more so, the number of effective tests to increase is bound to crash into the realities. With failed expectations, come disapointment, maybe an angered response, losing faith in the capabilities of the developers, and often calling it quits, way too early.

We’ll continue to discuss what else leaders can do in the next post.

 

This series deals with the implementation of a unit testing process in a team or across multiple teams in an organization. Posts in the series include:
Goals	Outcomes	Leading Indicators I	Leading Indicators II
Leading Indicators III	Leadership I	Leadership II	The plan

Part I was HUGE! Now, let’s look at broken builds. We want to see a decrease in their number over time.

This may sound a bit strange. Our CI is supposed to tell us if the build breaks, that’s its job. Isn’t having more of them a good thing?

Unsurprisingly, the answer is “it depends”.

As we want the earliest feedback, the answer is “yes, of course”, the CI system serves us well. However, if we don’t see a decrease in broken builds, that may mean the CI process is not working effectively. We should investigate.

CI PI

Let’s trace the steps leading to failing builds and see if we can improve our process.

Are all the tests passing locally? If not, we’re integrating code that fails tests into the trunk. If tests are not run locally, when they run in CI builds, they will probably fail too. That’s a big no-no. We may even find out the tests are not even run locally, and we’d want to improve on these behaviors.

If tests do run and pass locally before they are committed, there might be another problem. That may point to issues of isolation. If they pass locally, tests that depends on available resources in the local environments, find them there. But at the CI stage, they don’t and fail. More broken builds, indicate the team has not learned how to write isolated tests yet.

There might even be a bigger issue lurking.

Trust and accelerate feedback

We want to trust them on the CI environment, but since they “work on our machine” and not on the CI, these tests just got a  trust downgrade. This can have a weird counter effect on our way of running them.

Since results we trust run on the CI, and local runs are creating confusing results, we may stop running tests locally at all, and run them instead on the CI server making sure they run correctly there. When we do that, we make our feedback cycle longer, but more importantly, we risk the tests failing for the right reason, but holding the rest of the team hostage until they are fixed.

To get the right feedback early, we need to get back to running tests locally.

We want to increase the number of isolated tests, so they can be run locally, and can be trusted to fail on the CI server. Isolated unit or integration tests failing before committing is the first line of defense.

Then, we want to be able to run the non-isolated tests either locally or in a clean environment as we can manage. The point is to not commit code until we trust it. This may require changing available environments, modifying the tests to ensure cleanliness, pre-commit integration or any combination of those.

Can you believe all these improvement opportunities come from a single indicator? The deeper we dig, and more questions we ask, we can find opportunities for improving the process as a whole.

We’re not done yet.

This series deals with the implementation of a unit testing process in a team or across multiple teams in an organization. Posts in the series include:
Goals	Outcomes	Leading Indicators I	Leading Indicators II
Leading Indicators III	Leadership I	Leadership II	The plan

Now that we’ve talked about what we want to achieve, we better spread out some sensors. After all, we’re about to embark on a long and winding road. We want to know if we’re still going the right way, and if not, make a turn.

Leading indicators raise the red flag before the worst has happened. In our case, we don’t want to check in after six month and see no one’s written any tests for the last three months. We’d like to know sooner.

So what should we track?

If you’re thinking “coverage”, hold your horses. Coverage by itself is not a good progression metric. We sometimes mix simplicity of measurement with effectiveness, and coverage is one of those things.

Coverage as a number doesn’t have context – are we covering the right/risky/complex things? Are we exercising code but not asserting (my favorite gaming process)? Are we testing auto-generated code? Without the context, the coverage number has no applicable meaning.

Before we mandate tracking the entire software coverage, or setting a coverage goal, remember that while it is easy to track exercised code lines, the number doesn’t mean anything by itself.

So what do we look for?

The first metric to start tracking is the simplest one: Number of tests. (Actually two: tests written and test running).

Our first indicator if people are writing tests, is to count the tests. For that, we need to require a convention of test location, naming, etc. Guess what? These are needed also to run them as part of a CI build. Once everything is set up, we can count the tests where they count (pun!).

Then we want to look at the trend over time. The number should go up, as people add more tests. If the trend flatlines, we need to investigate.

One thing about this metric – if you never see a drop in the number of tests, something’s wrong. That probably means tests just stay there, and are not getting re-reviewed, replaced or deleted. In the short term, starting out, we want to see an upward trend. But over the long haul, code changes, and so do the tests. We want to see at least some fluctuations.

So what about that coverage?

Ok, we can measure coverage. We get that out of the box, right?

But we need to know what we’re measuring. Coverage means executed lines of code. So we can look at coverage (or lack of) as an indicator in the context we care for.

That could be any of the following:

• Important flows in the system
• Buggy code
• Code we return too over and over
• New component we want to cover

Or any other interesting bit. When we measure coverage, we want to see a trend of increasing coverage over these areas.

Now, how do you manage that? That requires more tricks, since we want to make sure we measure the right code, and the right tests. If the code architecture already supports it, it’s easy: 75% of a library for example.

If, however you want to measure coverage of a set of classes, excluding the other parts of the library, that requires more handling and management. Usually people don’t go there.

The funny thing is, the more specific you want to get, the less regular tools stop helping. And the broader numbers lose meaning.

By the way, the coverage metric should go away once you get to sufficient coverage. Once again, it’s not a number, but what we want to achieve – stability over time, or regression coverage. We can stop measuring then (and may be look at other areas).

Ok, we’ll continue the indicators discussion in the next post.