How to catch up on test automation

(this entry is now available as an article on the scrum alliance site as well)

The test automation problem

Many companies with existing legacy code bases bump into a huge impediment when they want to get agile: lack of test automation.

Without test automation it is very hard to make changes in the system, because things break without anybody noticing. When the new release goes live, the defects are discovered by the real users, causing embarrassment and expensive hotfixing. Or even worse, a chain of hotfixes because each hotfix introduces new unanticipated defects.

This makes the team terribly afraid to change code, and therefore relucant to improve the design of the code, which leads to a downward spiral of worse and worse code as the system grows.

What to do about it

Your main options in this case are:

1. Ignore the problem. Let the system decline into entropy death, and hope that nobody needs it by then.
2. Rebuild the system from scratch using test-driven development (TDD) to ensure good test coverage.
3. Start a separate test automation project where a dedicated team improves the test coverage for the system until it is adequate
4. Let the team improve test coverage a little bit each sprint.

Guess which approach usually works best? Yep, the last one – improve test coverage a little bit each sprint. At least in my experience.

The third option may sound tempting, but it is risky. Who’s going to do the test automation? A separate team? If so, does that mean the other developers don’t need to learn how to automate tests? That’s a problem. Or is the whole team doing the test automation project? In that case their velocity (from a business perspective) is 0 until they are done. So when are they done? When does test automation “end”?

No, let’s get back to the fourth option. Improve test coverage a little bit each sprint. So, how to do that in practice?

How to improve test coverage a little bit each sprint

Here’s an approach that I like. In summary:

1. List your test cases
2. Classify each test by risk, how expensive it is to do manually, and how expensive it is to automate
3. Sort the list in priority order
4. Automate a few tests each sprint, starting from the highest priority.

Step 1: List your test cases

Think about how you test your system today. Brainstorm a list of your most important test cases. The ones that you already execute manually today, or wish you had time to execute. Here’s an example from a hypothetical online banking system:

Change skin

Security alert

See transaction history

Block account

Add new user

Sort query results

Deposit cash

Validate transfer

Step 2: Classify each test

First classify your test cases by risk. Look at your list of tests. Ignore the cost of manual testing for the moment. Now what if you could throw away half of the tests, and never execute them? Which tests would you keep? This factor is a combination of the probability of failure and cost of failure.

Highlight the risky tests, the ones that keep you awake at night.

Test case	Risk
Change skin
Security alert	x
See transaction history
Block account	x
Add new user
Sort query results
Deposit cash	x
Validate transfer	x

Now think about how long each test takes to execute manually. Which half of the tests take the longest? Highlight those.

Test case	Risk	Manual test cost
Change skin
Security alert	x
See transaction history		x
Block account	x	x
Add new user
Sort query results		x
Deposit cash	x
Validate transfer	x	x

Finally, think about how much work it is to write a automation scripts for each test. Highlight the most expensive half.

Test case	Risk	Manual test cost	Automation cost
Change skin			x
Security alert	x		x
See transaction history		x
Block account	x	x
Add new user
Sort query results		x	x
Deposit cash	x
Validate transfer	x	x	x

Step 3: Sort the list in priority order

So, which test do you think we should automate first? Should we automate “Change skin” which is low-risk, easy to test manually, and difficult to automate? Or should we automate “Block account” which is high risk, difficult to test manually, and easy to automate? That’s a fairly easy decision.

But here’s a more difficult decision. Should we automate “Validate transfer” which is high-risk, hard to test manually, and hard to automate? Or should we automate “Deposit cash” which also is high-risk, but easy to test manually and easy to automate? That decision is context dependent.

You basically need to make three decisions:

• Which do you automate first? The high risk test that is easy to test manually, or the low risk test that is difficult to test manually?
• Which do you automate first? The test that is easy to do manually and easy to automate, or the test that is hard to do manually and hard to automate?
• Which do you automate first? The high risk test that is hard to automate, or the low risk test that is easy to automate?

Those decisions will give you a prioritization of your categories, which in turn lets you sort your list of test cases by priority. In my example I decided to prioritize manual cost first, then risk, then automation cost.

Test case	Risk	Manual test cost	Automation cost
Block account	x	x
Validate transfer	x	x	x
See transaction history		x
Sort query results		x	x
Deposit cash	x
Security alert	x		x
Add new user
Change skin			x

So that’s it! A prioritized backlog of test automation stories.

You could of course also invent some kind of calculation algorithm. A simple such algorithm is that each highlighted cell = 1 point. Then you just add upp each row and sort. Or just sort the list manually using gut feel.

You could also use a more specific unit for each category, if my simple binary scheme isn’t sufficient.

Test case	Risk	Manual test cost (man-hours)	Automation cost (story points)
Block account	high	5 hrs	0.5 sp
Validate transfer	high	3 hrs	5 sp
See transaction history	medium	3 hrs	1 sp
Sort query results	medium	2 hrs	8 sp
Deposit cash	high	1.5 hr	1 sp
Security alert	high	1 hr	13 sp
Add new user	low	0.5 hr	3 sp
Change skin	low	0.5 hr	20 sp

Remember though that our goal for the moment is just to prioritize the list. If you can do that with a simple and crude categorization scheme then there’s no need to complicate things right? Analysis is useful but over-analysis is just a waste of time.

Step 4 – Automate a few tests each sprint

Irregardless of the stuff above, each new product backlog story should include test automation at the story level. That’s the XP practice known as “customer acceptance tests”. Not doing that is what got your system into this mess in the first place.

But in addition to implementing new stories, we want to spend some time automating old test cases for other previously existing stories. So how much time do we spend? The team needs to negotiate that with the product owner. The agreement will typically take on one of the following forms:

• “Each sprint we will implement one test automation story”
• “Each sprint we will implement up to 10 story points of test automation stories”
• “Each sprint we will spend about 10% of our time implementing test automation stories”
• “Each sprint we will finish the product backlog stories first, and then spend the remainder of the time (if any) implementing test automation stories”
• “The product owner will merge the test automation stories into the overall product backlog, and the team will treat them just like any other story.”

The exact form of the agreement doesn’t matter. You can change it every sprint if you like. The important thing is that the test automation debt is being gradually repaid, step by step.

After finishing half the stories on your test automation backlog you might decide that “hey, we’ve paid back enough debt now! Let’s just skip the rest of the old test cases, they’re not worth automating anyway”, and dump the rest. Congratulations!

So this solves the problem?

Wishful thinking. No, this pattern doesn’t magically solve your test automation problem. But it makes the problem easier to approach :o)