Henrik Kniberg

Henrik Kniberg

I debug, refactor, and optimize IT companies. And jam alot too.

How to catch up on test automation

(this entry is now available as an article on the scrum alliance site as well)

The test automation problem

Many companies with existing legacy code bases bump into a huge impediment when they want to get agile: lack of test automation.

Without test automation it is very hard to make changes in the system, because things break without anybody noticing. When the new release goes live, the defects are discovered by the real users, causing embarrassment and expensive hotfixing. Or even worse, a chain of hotfixes because each hotfix introduces new unanticipated defects.

This makes the team terribly afraid to change code, and therefore relucant to improve the design of the code, which leads to a downward spiral of worse and worse code as the system grows.

What to do about it

Your main options in this case are:

  1. Ignore the problem. Let the system decline into entropy death, and hope that nobody needs it by then.
  2. Rebuild the system from scratch using test-driven development (TDD) to ensure good test coverage.
  3. Start a separate test automation project where a dedicated team improves the test coverage for the system until it is adequate
  4. Let the team improve test coverage a little bit each sprint.

Guess which approach usually works best? Yep, the last one – improve test coverage a little bit each sprint. At least in my experience.

The third option may sound tempting, but it is risky. Who’s going to do the test automation? A separate team? If so, does that mean the other developers don’t need to learn how to automate tests? That’s a problem. Or is the whole team doing the test automation project? In that case their velocity (from a business perspective) is 0 until they are done. So when are they done? When does test automation “end”?

No, let’s get back to the fourth option. Improve test coverage a little bit each sprint. So, how to do that in practice?

How to improve test coverage a little bit each sprint

Here’s an approach that I like. In summary:

  1. List your test cases
  2. Classify each test by risk, how expensive it is to do manually, and how expensive it is to automate
  3. Sort the list in priority order
  4. Automate a few tests each sprint, starting from the highest priority.

Step 1: List your test cases

Think about how you test your system today. Brainstorm a list of your most important test cases. The ones that you already execute manually today, or wish you had time to execute. Here’s an example from a hypothetical online banking system:

Change skin
Security alert
See transaction history
Block account
Add new user
Sort query results
Deposit cash
Validate transfer

Step 2: Classify each test

First classify your test cases by risk. Look at your list of tests. Ignore the cost of manual testing for the moment. Now what if you could throw away half of the tests, and never execute them? Which tests would you keep? This factor is a combination of the probability of failure and cost of failure.

Highlight the risky tests, the ones that keep you awake at night.

Test case Risk
Change skin
Security alert  x
See transaction history
Block account  x
Add new user
Sort query results
Deposit cash  x
Validate transfer  x

Now think about how long each test takes to execute manually. Which half of the tests take the longest? Highlight those.

Test case Risk Manual test cost
Change skin
Security alert  x
See transaction history  x
Block account  x x
Add new user
Sort query results  x
Deposit cash  x
Validate transfer  x  x

Finally, think about how much work it is to write a automation scripts for each test. Highlight the most expensive half.

Test case Risk Manual test cost Automation cost
Change skin  x
Security alert  x  x
See transaction history  x
Block account  x  x
Add new user
Sort query results  x  x
Deposit cash  x
Validate transfer x  x  x

Step 3: Sort the list in priority order

So, which test do you think we should automate first? Should we automate “Change skin” which is low-risk, easy to test manually, and difficult to automate? Or should we automate “Block account” which is high risk, difficult to test manually, and easy to automate? That’s a fairly easy decision.

But here’s a more difficult decision. Should we automate “Validate transfer” which is high-risk, hard to test manually, and hard to automate? Or should we automate “Deposit cash” which also is high-risk, but easy to test manually and easy to automate? That decision is context dependent.

You basically need to make three decisions:

  • Which do you automate first? The high risk test that is easy to test manually, or the low risk test that is difficult to test manually?
  • Which do you automate first? The test that is easy to do manually and easy to automate, or the test that is hard to do manually and hard to automate?
  • Which do you automate first? The high risk test that is hard to automate, or the low risk test that is easy to automate?

Those decisions will give you a prioritization of your categories, which in turn lets you sort your list of test cases by priority. In my example I decided to prioritize manual cost first, then risk, then automation cost.

Test case Risk Manual test cost Automation cost
Block account  x  x
Validate transfer  x  x  x
See transaction history  x
Sort query results  x  x
Deposit cash  x
Security alert  x  x
Add new user
Change skin  x

So that’s it! A prioritized backlog of test automation stories.

You could of course also invent some kind of calculation algorithm. A simple such algorithm is that each highlighted cell = 1 point. Then you just add upp each row and sort. Or just sort the list manually using gut feel.

You could also use a more specific unit for each category, if my simple binary scheme isn’t sufficient.

Test case Risk
Manual test cost
Automation cost
(story points)
Block account high 5 hrs 0.5 sp
Validate transfer high 3 hrs 5 sp
See transaction history medium 3 hrs 1 sp
Sort query results medium 2 hrs 8 sp
Deposit cash high 1.5 hr 1 sp
Security alert high 1 hr 13 sp
Add new user low 0.5 hr 3 sp
Change skin low 0.5 hr 20 sp

Remember though that our goal for the moment is just to prioritize the list. If you can do that with a simple and crude categorization scheme then there’s no need to complicate things right? Analysis is useful but over-analysis is just a waste of time.

Step 4 – Automate a few tests each sprint

Irregardless of the stuff above, each new product backlog story should include test automation at the story level. That’s the XP practice known as “customer acceptance tests”. Not doing that is what got your system into this mess in the first place.

But in addition to implementing new stories, we want to spend some time automating old test cases for other previously existing stories. So how much time do we spend? The team needs to negotiate that with the product owner. The agreement will typically take on one of the following forms:

  • “Each sprint we will implement one test automation story”
  • “Each sprint we will implement up to 10 story points of test automation stories”
  • “Each sprint we will spend about 10% of our time implementing test automation stories”
  • “Each sprint we will finish the product backlog stories first, and then spend the remainder of the time (if any) implementing test automation stories”
  • “The product owner will merge the test automation stories into the overall product backlog, and the team will treat them just like any other story.”

The exact form of the agreement doesn’t matter. You can change it every sprint if you like. The important thing is that the test automation debt is being gradually repaid, step by step.

After finishing half the stories on your test automation backlog you might decide that “hey, we’ve paid back enough debt now! Let’s just skip the rest of the old test cases, they’re not worth automating anyway”, and dump the rest. Congratulations!

So this solves the problem?

Wishful thinking. No, this pattern doesn’t magically solve your test automation problem. But it makes the problem easier to approach :o)

Scrum success stories

Scrum isn’t the only agile method (see my article Scrum and XP work well together), but seems to be one of the best ways to get started. As I mentioned in Failing with Scrum, Scrum is no silver bullet. It doesn’t guarantee success, but it improves the odds.

Over the past few years I’ve been involved in dozens of Scrum projects, directly or indirectly. Interestingly enough, every single case that I can remember has been successful (maybe I just have selective memory…)! The client has been happy with the process and wants to continue using Scrum in future projects.

In most cases Scrum has resulted in immense improvements. Based on interviews, most improvements are within the following areas:

  • business focus
  • productivity
  • product quality
  • team motivation
  • transparancy & knowledge spread

In some cases Scrum has resulted in only minor improvements, i.e. the client is only slightly better off than before.

In one case a client aborted a project shortly after introducing Scrum, but they considered it a success since Scrum helped them discover early that the project was doomed to fail (described in more detail in Failing with Scrum).

No clients so far have gone back to their previous process. On the contrary, every company that I’ve worked with has decided to ramp up and use Scrum on other projects as well. Several have decided to go "company wide" and introduce Scrum, lean and agile techniques throughout all layers in the company.

Here’s a quote from an email I received today (from one of my readers, not a client):

At this point I can already proudly mention, that the project has been just like in a fairy tale. We were able to finish the implementation well ahead of schedule and with a quality that has surprised many. One good indicator of successful first-ever-Scrum-project is that today all of our project managers have either completed or signed up for a Certified Scrum Master course. I bet we’ll soon have a few more Scrum projects running 😉

Best part any way is, that the developers seem to love Scrum (just as you mentioned).

Thanks again for writing the book, it helped us a lot!

My book Scrum and XP from the trenches describes concrete agile techniques used in a handful of different projects within a single company. Some of the techniques failed, but we inspected and adapted and each project was a success, even those that seemed very bleak and uncertain at the outset.

After publishing the book I’ve received literally hundreds of emails from people using Scrum and/or XP, including many companies that were inspired to give agile software development a shot after reading the book. Very good to hear!

The book was not intended to be a "manual" or "cook book" for Scrum and XP – it was intended to be a case study with detailed examples. Although I stated that quite clearly in my disclaimer, many companies have nevertheless used it as a cook book and implemented Scrum exactly as described in the book. To my surprise, they have done well! Even in totally different contexts. The context of the book is a gaming software company, but I’ve received reports from companies building things such as flight control software and embedded software using the same process and techniques, and it has worked well!

I still don’t recommend using my book as a manual. I’m still dreading the email "Hi Henrik, we found your book and have implemented everything as described. And now the project went to hell and we’re going to sue you! Our software caused the nuclear plant to explode. See you in court!". But it hasn’t happened yet…

After all this positive feedback, and a complete lack of negative feedback, I’ve made some new conclusions:

  • Scrum appears to work very well in a very broad range of contexts
  • Some implementation patterns (= concrete techniques) are quite universal and can be applied in almost any context. I used to think they were more context dependent. For example:
    • Physical taskboards
    • Planning poker
    • Combining Scrum and XP
    • … and many more
  • Copying another company’s implementation is a good idea. But it should be seen as a starting point only. A code example. A template from which to evolve.
  • Make sure you have a coach. Doesn’t necessarily need to be an external coach, just  make sure you have somebody in the building who’s done this stuff before.

Failing with Scrum

There are plenty of Scrum success stories out there, but not so many failure stories. That gives rise to a healthy scepticism. Where are the failure stories? Obviously there must be failures! Or?

Introducing Schlum

Well, here’s another process for you. It’s called Schlum, I just invented it. It has two principals:

  • Schlum principle #1: Do the right thing
  • Schlum principle #2: Don’t do the wrong thing

This process has a great track record! I can’t remember any single project failing with this method! Wait, come to think of it, one did actually fail. But that’s because they did it wrong. They mixed up the words "right" and "wrong"! Once I pointed that out to them things got better :o)

Failing with Scrum & Schlum

Why is it so hard to fail with Schlum? Because it is a process framework. It addresses the most fundamental bits, and leaves all the scary details to you. Slippery isn’t it?

Scrum is sort of similar. Most of Scrum translates to common sense. Deliver working code early and often. Work in short cycles to minimize risk and maximize feedback. Do the most important stuff first in case we run out of time. Discuss and improve the process on a regular basis. Show the customer stuff as early as possible. 

Not always easy, but certainly desirable! Who wouldn’t want a process like this? When would this approach in itself cause a project to fail?

But then there are lots of other really important things that Scrum doesn’t address. How do you test? How do you decide who should be on what team? How do you synchronize diverging code bases? How do you find the most representative customer representative? Etc, etc.

THOSE are the failure points. You can really get things wrong if there is no customer to talk to, to name one example.

A process framework isn’t enough – you need the patterns as well

Scrum is a process framework. It is a process for creating a process. It contains slightly more meat than Schlum, but is still very bare-bones.

Scrum provides feedback loops and control points that make you continuously rub against the weak points of your current process, and thereby help you inspect & adapt your process to your specific context.

Now, a process framework is in itself not very useful.

Take Schlum for example. What more do you need? Just do Schlum and you’ll be fine! Uh, what did you say? How to do the right thing? Who decides what the right thing is? Have you ever heard an expensive consultant say "Oh those are implementation details, it is up to you". Not a very useful answer. A better answer is "Oh, those are implementation details – here are some examples of how those issues have been addressed in other similar contexts".

I’m talking about patterns. There are lots of books and sites on organizational patterns. Patterns are most useful when they are accompanied with an example describing how a particular problem was solved in a particular context. My book Scrum and XP from the Trenches is full of patterns and anti-patterns and examples of how they were implemented, although it isn’t really organized as a formal pattern repository.

A failure story. Or?

One of my clients had decided to start using Scrum for a project. They had a grandiose vision for what they were going to build and wanted to get started as quickly as possible.

I asked them to wait, and instead start by creating a product backlog, and get it roughly estimated by the team (actually it was just a few representatives of the team-to-come). They wanted to start by doing prestudies and building platforms. I asked them instead to fully implement the first few stories in "prototype" mode, just to get something to work.  The goal of the first sprint would be to learn enough about the first few stories in order to make better estimates for the rest of the product backlog, and learn about the team’s velocity.

A few weeks after helping them get started I called back to ask how they were doing. They had canceled the project! My company lost some business there, as we were supposed to be involved in the development.

Hmmmm… what happened here? They were going to start a new cool project. In fact they were already in the middle of the pre-study. And then they decided to start using Scrum. And a few weeks later they canceled the project.

So – did Scrum fail?

No. Scrum succeeded. The project was canceled because it turned out to be unclear, overambitious and unrealistic. The wanted to build too much stuff in too short time. Scrum was the messenger that helped them fail early rather than late. Without Scrum they may well have commited significant resources to the project, only to discover several months and many bucks later that it was doomed to fail. Nobody outside the project needed to convince them, they discovered it themselves.

Although we lost some business on that one project, we won a very satisfied and faithful customer who now hires us regularly for other projects. They are also continuously expanding their use of Scrum.

This type of story is very common. It is an example of a successful process and a failed project. There will always be projects that fail, the trick is to get them to fail as early as possible when they do.

It’s easy to confuse project success/failure with process success/failure.

  • A project may succeed because of a great process
  • A project may succeed despite a lousy process
  • A project may fail because of a lousy process
  • A project may fail despite a great process

So when does Scrum itself fail?

Scrum fails when it is misapplied. You don’t need to get it right from the start, but you do need to continuously inspect and adapt the process.

Maybe there are projects that have failed because of Scrum, i.e. where Scrum was implemented correctly and actually caused the project to fail. I just haven’t seen a convincing case yet, let me know if you have one. Usually failure stories travel faster than success stories, so if it Scrum failures were common I’m sure they would be well spread.

There is a paradox, an unfortunate rhetoric that some Scrum trainers use. I accidentally use it myself sometimes: "If Scrum failed, you did it wrong". This is a tautology, a meaningless statement. You can use that type of statement to justify anything – "If waterfall failed, you did it wrong". It is like saying "Scrum can’t fail, because if Scrum fails it isn’t Scrum". Note that my statement "Scrum fails when it is misapplied" isn’t really the same.

Scrum is just a framework. You have to fill out the blanks yourself, find the patterns and experiment with them. Sometimes you will get it wrong. Sometimes the corporate culture will get in the way and stop you from making the necessary changes.

Finding Scrum patterns is easy. Just read web sites and books on Scrum and agile software development.

The hard part is knowing when to apply which pattern. And how do you learn that?

  1. Experience – try the patterns out in different situations, see what works and what doesn’t.
  2. Talk to others that have the experience.
  3. Read books & papers written by people that have the experience. 

Did I mention the word "experience"?

Some patterns are quite universally applicable, for example the use of physical taskboards. Other patterns are more contexts-dependent, for example how to divide people into teams.

How to minimize the risk of process failure

  • Learn
  • Create & utilize feedback loops

Read books & articles, attend conferences, talk to colleagues, participate in online forums such as scrumdevelopment@yahoogroups.com (LOTS of patterns to be found there, but takes some digging!).  Experiment with different ways of implementing Scrum, but make sure you don’t accidentally break the critical feedback loops of Scrum – the daily scrum and the sprint review.

And remember that the improvement process never ends – you can always make things better! And besides, your context will evolve so the process has to evolve with it.


Guess what. It takes time to learn! It takes time to create & utilize feedback loops. This is typically the role of the Scrum Master.

If you tell me "I don’t have time" then I will tell that you are probably wasting time! You are probably fruitlessly banging your blunt axe against the tree, instead of taking a few minutes to sharpen it. You are probably running with the bicycle on your back, instead of stopping a few seconds to get on it.

Take the time. Reprioritize. If nobody on your project has time to learn about process patterns, and nobody has time to create & utilize feedback loops in your process, there is a high likelihood that you are dooming your project to failure – and working very hard to do it.

So if Scrum itself rarely fails, is Scrum a Silver Bullet?

Nope. Scrum doesn’t guarantee success. It just improves the odds :o)