Tag Archives: testing

Automated testing is never enough

Posted on by

In the pursuit to automate testing to create faster feedback loops and build quality in, some teams forget the value of manual testing. In my experience, without manual testing (as well) we are toast.

read more »

Security Test-Driven Development – Spreading the STDD-virus

Posted on by

Agile development with short release cycles have been here for a while now. Most of us want fast feedback loops and many even Continuous Delivery with changes in production software everyday. However, most of us also want secure software and the question is: Can security engineering keep up the pace? A fast feedback that your production website has been hacked is not so nice.

Security is a quality attribute of your software, just like performance. If you don’t want to be surprised by bad performance in production, what do you do? You test and design for it of course and you preferably do so continuously from the start.

In my experience, the same however cannot be said of security. It is very often relegated to a once a year penetration-test activity. Not really an agile way of working is it? Not a secure one either since untested software is released as often as everyday. There must be a better way of working which allows us to both work in an agile way and to verify security on the way.

In the security field people like Gary McGraw have long been advocating ways of “Building Security In”. The Microsoft MVP Troy Hunt also proposes that you should “Hack yourself first”, instead of just waiting for the pentesters. Shouldn’t it be possible to weave these security activities into the process the same way as it is possible with normal testing activities using TDD? Indeed I, as well others believe it is so. Let’s look at how small extensions to an agile process can work in this direction.

Extending Sprint planning to deal with security

To start off you must first know what the requirements are. In a normal agile project this is done by eliciting User Stories from the customer or the Product Owner.

Let’s take an example of an online e-Commerce site. A User Story might be “As a customer I want to be able to add a review of a product so that information about products can be shared between customers”.

This works very well for traditional functional requirements, but for non-functional requirements a little extra thought is needed. In the case of security requirements it is often useful to state a requirement in a scenario that should NOT happen. In our case we shall call these scenarios “Abuser Stories”. These stories are non-technical descriptions of bad things you want to make sure you avoid. An Abuser story for this site might be:

“An attacker uses the Review Product-function to spread malicious Javascript”. Another might be: “An attacker abuses the Review Product-function to gain unlimited access to the database”.
A Product Owner might not be able to come up with these stories himself, but might need the help of a security engineer to help him with finding these threat scenarios.

read more »

What is an Agile Tester? Slides from my Sri Lanka talk.

Posted on by

Here are the slides for my talk What is an agile tester from the Colombo Agile Conference in Sri Lanka.

What is an agile tester

read more »

Stop-the-line spoken word performance

Posted on by

Agila Sverige 2012 höll jag min första ignite i form av en Majakovski- och Bob Dylan-inspirerad Spoken World performance om hur vi på Polopoly skapade kvalitet genom extremt fokus på automatiserade tester och en stoppa-bandet-kultur.

Förra veckan fyllde konsultbolaget Adaptiv 5 år och firade genom att låte en utvald skara Agila Sverige-talare reprisera sina alster. Jag fick äran att stå för en liten del av middagsunderhållingen. Här är uppläsningen:

Interview with Peter Antman on software development

Posted on by

In May this year I was interviewed at the SmartBear MeetUI user conference in Stockholm about my background as a journalist, Linux and open source, software development in general and testing more specifically. Here’s a post containing the video and interviews with the other speakers at the conference.

Still not automating tests? Here’s why you should (again)!

Posted on by

The other day I read a blog by Uncle Bob. It more or less stated that no matter what situation you are in, writing automated tests will make you go faster. Ok, this is old news I thought, until I checked Uncle Bob’s tweets. A fair amount of people argued against this statement, and that surprised me!


Join me at the campfire!

So I started thinking about why there still are fellow software developers that doesn’t believe in automated testing? Have they not seen them in action and understood what they are for? Please, gather around the campfire, and I will tell you one, just one, of my war stories, and then I will tell you why also you should write automated tests!

read more »

Always Fix Broken Windows

Posted on by

I keep a close watch on these tests of mine
I keep my Jenkins open all the time
I see a defect coming down the line
Because you’re mine, I stop the line

A zero bug policy is the only valid way to look at quality, just like there should never be any broken windows in your neighbourhood. Here is the third part of my three-part series on building the quality in on the SmartBear blog.

Think of old bicycles in cities. A bike is left tied up on the sidewalk for a while. Someone steals the ringing bell. Left in that shape in the same place for long enough the bike will magically fall apart, as if rotting.

In 1982, the social scientists James Q. Wilson and George L. Kelling formulated a sociological theory from observations like these called “The Broken Windows.” A landscape, they meant, signals the type of social norms prevalent in the local society. A broken window left broken says: “it’s okay to break windows.” A broken bicycle sends the same signal.

Read the rest of the blog at SmartBear.

Stop the Line – Build Quality In with Incremental Compilation

Posted on by

We in the software industry are still far behind when it comes to automated quality checks. Toyoda Sakichi for example invented the automated loom with stop the line capability almost 100 years ago. I write more about that in my first blog in a three-part series on building the quality in on the SmartBear blog.

It may seem old school, but every now and then I’ll start pristine Emacs and hack away on a piece of Java. I mainly do this for small stuff, since starting or setting up Eclipse takes a little bit too long for such minor projects. Even so, I don’t do this as often as I used to.

Why? I guess you know the answer: Nowadays, even the thought of not having instant feedback on syntactical and semantical correctness makes me shudder. It’s like coding in the dark. Incremental compilation is one of those great innovations that have increased the quality of our work.

Read the rest of the blog at SmartBear.

Country Ambassador for Agile Testing Days 2012

Posted on by

A while ago I was asked to become one of the Swedish country ambassadors for the Agile Testing Days 2012 conference. I said yes, because I think it’s a great conference. As country ambassador, I help in promoting the conference. I chose to do it, because I think it’s a good conference and I already recommend it to my friends.
read more »

How to catch up on test automation

Posted on by

(this entry is now available as an article on the scrum alliance site as well)

The test automation problem

Many companies with existing legacy code bases bump into a huge impediment when they want to get agile: lack of test automation.

Without test automation it is very hard to make changes in the system, because things break without anybody noticing. When the new release goes live, the defects are discovered by the real users, causing embarrassment and expensive hotfixing. Or even worse, a chain of hotfixes because each hotfix introduces new unanticipated defects.

This makes the team terribly afraid to change code, and therefore relucant to improve the design of the code, which leads to a downward spiral of worse and worse code as the system grows.

What to do about it

Your main options in this case are:

  1. Ignore the problem. Let the system decline into entropy death, and hope that nobody needs it by then.
  2. Rebuild the system from scratch using test-driven development (TDD) to ensure good test coverage.
  3. Start a separate test automation project where a dedicated team improves the test coverage for the system until it is adequate
  4. Let the team improve test coverage a little bit each sprint.

Guess which approach usually works best? Yep, the last one – improve test coverage a little bit each sprint. At least in my experience.

The third option may sound tempting, but it is risky. Who’s going to do the test automation? A separate team? If so, does that mean the other developers don’t need to learn how to automate tests? That’s a problem. Or is the whole team doing the test automation project? In that case their velocity (from a business perspective) is 0 until they are done. So when are they done? When does test automation “end”?

No, let’s get back to the fourth option. Improve test coverage a little bit each sprint. So, how to do that in practice?

How to improve test coverage a little bit each sprint

Here’s an approach that I like. In summary:

  1. List your test cases
  2. Classify each test by risk, how expensive it is to do manually, and how expensive it is to automate
  3. Sort the list in priority order
  4. Automate a few tests each sprint, starting from the highest priority.

Step 1: List your test cases

Think about how you test your system today. Brainstorm a list of your most important test cases. The ones that you already execute manually today, or wish you had time to execute. Here’s an example from a hypothetical online banking system:

Change skin
Security alert
See transaction history
Block account
Add new user
Sort query results
Deposit cash
Validate transfer

Step 2: Classify each test

First classify your test cases by risk. Look at your list of tests. Ignore the cost of manual testing for the moment. Now what if you could throw away half of the tests, and never execute them? Which tests would you keep? This factor is a combination of the probability of failure and cost of failure.

Highlight the risky tests, the ones that keep you awake at night.

Test case Risk
Change skin
Security alert  x
See transaction history
Block account  x
Add new user
Sort query results
Deposit cash  x
Validate transfer  x

Now think about how long each test takes to execute manually. Which half of the tests take the longest? Highlight those.

Test case Risk Manual test cost
Change skin
Security alert  x
See transaction history  x
Block account  x x
Add new user
Sort query results  x
Deposit cash  x
Validate transfer  x  x

Finally, think about how much work it is to write a automation scripts for each test. Highlight the most expensive half.

Test case Risk Manual test cost Automation cost
Change skin  x
Security alert  x  x
See transaction history  x
Block account  x  x
Add new user
Sort query results  x  x
Deposit cash  x
Validate transfer x  x  x

Step 3: Sort the list in priority order

So, which test do you think we should automate first? Should we automate “Change skin” which is low-risk, easy to test manually, and difficult to automate? Or should we automate “Block account” which is high risk, difficult to test manually, and easy to automate? That’s a fairly easy decision.

But here’s a more difficult decision. Should we automate “Validate transfer” which is high-risk, hard to test manually, and hard to automate? Or should we automate “Deposit cash” which also is high-risk, but easy to test manually and easy to automate? That decision is context dependent.

You basically need to make three decisions:

  • Which do you automate first? The high risk test that is easy to test manually, or the low risk test that is difficult to test manually?
  • Which do you automate first? The test that is easy to do manually and easy to automate, or the test that is hard to do manually and hard to automate?
  • Which do you automate first? The high risk test that is hard to automate, or the low risk test that is easy to automate?

Those decisions will give you a prioritization of your categories, which in turn lets you sort your list of test cases by priority. In my example I decided to prioritize manual cost first, then risk, then automation cost.

Test case Risk Manual test cost Automation cost
Block account  x  x
Validate transfer  x  x  x
See transaction history  x
Sort query results  x  x
Deposit cash  x
Security alert  x  x
Add new user
Change skin  x

So that’s it! A prioritized backlog of test automation stories.

You could of course also invent some kind of calculation algorithm. A simple such algorithm is that each highlighted cell = 1 point. Then you just add upp each row and sort. Or just sort the list manually using gut feel.

You could also use a more specific unit for each category, if my simple binary scheme isn’t sufficient.

Test case Risk
Manual test cost
Automation cost
(story points)
Block account high 5 hrs 0.5 sp
Validate transfer high 3 hrs 5 sp
See transaction history medium 3 hrs 1 sp
Sort query results medium 2 hrs 8 sp
Deposit cash high 1.5 hr 1 sp
Security alert high 1 hr 13 sp
Add new user low 0.5 hr 3 sp
Change skin low 0.5 hr 20 sp

Remember though that our goal for the moment is just to prioritize the list. If you can do that with a simple and crude categorization scheme then there’s no need to complicate things right? Analysis is useful but over-analysis is just a waste of time.

Step 4 – Automate a few tests each sprint

Irregardless of the stuff above, each new product backlog story should include test automation at the story level. That’s the XP practice known as “customer acceptance tests”. Not doing that is what got your system into this mess in the first place.

But in addition to implementing new stories, we want to spend some time automating old test cases for other previously existing stories. So how much time do we spend? The team needs to negotiate that with the product owner. The agreement will typically take on one of the following forms:

  • “Each sprint we will implement one test automation story”
  • “Each sprint we will implement up to 10 story points of test automation stories”
  • “Each sprint we will spend about 10% of our time implementing test automation stories”
  • “Each sprint we will finish the product backlog stories first, and then spend the remainder of the time (if any) implementing test automation stories”
  • “The product owner will merge the test automation stories into the overall product backlog, and the team will treat them just like any other story.”

The exact form of the agreement doesn’t matter. You can change it every sprint if you like. The important thing is that the test automation debt is being gradually repaid, step by step.

After finishing half the stories on your test automation backlog you might decide that “hey, we’ve paid back enough debt now! Let’s just skip the rest of the old test cases, they’re not worth automating anyway”, and dump the rest. Congratulations!

So this solves the problem?

Wishful thinking. No, this pattern doesn’t magically solve your test automation problem. But it makes the problem easier to approach :o)

Manifest for the Agile Tester

Posted on by

I recently spoke at a conference for testers arranged by SAST (Swedish Association of Software Testers) on the topic of Agile software development. Over 150 testers turned up, breaking all previous records for that association! I’m glad to see that you testers are interested in this stuff!

Anyway, trying to figure out a good opening statement for this conference I found the following angle that I’m pretty happy with afterwards, in a smug sort of way :o)

Don’t remember exactly what I said, but it was something like this:

"Is there anybody here who works with test?"

(Most hands raised. Mock surprised look on my face. "Wow, that many?" Sorry, couldn’t resist)

"Well I have news for you. In a truly agile project there is no role called Tester. There is no team called Test Team. And there is no project phase called Testing"

(A few seconds pause to take in the deathly silence and the look of 150 worried faces)

"Why is this? Is testing not important in agile projects? Have we magically done away with the need for testing?"

"Certainly not. Quite the contrary. In agile projects, testing is considered to be far too important to be confined to a single role, a single team, or a single project phase. Having a role called Tester implies that others don’t need to test. In an agile project almost everyone tests. Having a project phase called Testing implies that testing isn’t in the other phases. In an agile project testing is done almost all the time."

"Now I bet you are thinking but programmers suck at testing! Are there any programmers here?"

(a couple of hands up)

"Are you lousy at testing?"

(He nods and giggles a bit. Sorry, couldn’t resist again.)

"Well, I wouldn’t say programmers are bad at testing. It’s just that they test in a different way. In agile projects they pair program, for one. Yes, contrary to what most people think, pair programming is a testing technique (why would anyone think pair programming is a programming technique? :o). It is code review in real-time, the most efficient way of testing. A bug found while writing code can usually be fixed in a few man-seconds. Compare this to the man-hours or man-days of time needed to fix bugs discovered later on in the project, when you have no idea which part of the code caused the bug."

"The customer tests as well. He tests by trying out the system early and often, giving feedback on things such as usability and business value. Product owners test by helping the team define acceptance test criteria. Developers test by writing automated test code. Code who’s only purpose is to verify the correctness of other code. Not only that, they write the test code first, before they write the code that is to be tested! This is known as TDD, or Test-Driven Development, a tool who’s value is almost impossible to overstate."

"And then, of course, there’s you – the one they used to call The Tester. You are no longer The Tester. You are a fully integrated team member, and will share all of its successes and failures. You will do whatever is most important to your project at any given time – whether it is related to test or not. Welcome in!"

"Testing is your expertise, but on an agile team all of your talents will come to use, even talents nobody knew you had. You might help ensure that requirements (called User Stories) are testable from the very beginning. You might pair-program with developers to help them write better test code. You might define acceptance criteria together with customers. You might serve coffee. You might code. You might set up test environments. And last, but not least, you will test, just like everybody else"

"Since the programmers have automated all the boring, repetetive tests, you are free to focus on the hard stuff. The stuff that is difficult to automate. Exploratory testing. Usability testing. Performance test scripts. Figuring out those tricky test cases that only a veteran like you could come up with. And when you find bugs, you will help developers write code to demonstrate the bug and ensure it never happens again. You will coach the developers, teach them to think about things like boundary value analysis and equivalence partitioning and model-based testing and whatever other tricks you have up your sleeve. Your job is not to find bugs, your job is to help the team prevent them! The team is your team."

"Agile teams are cross functional, they are made up of generalizing specialists. People who are experts at certain areas, but have basic knowledge of a whole bunch of other areas as well. A team of generalizing specialists is extremely fast at learning and adapting."

"If you are a generalizing specialist already, then I congratulate you. If not, I hope you like to learn!"

"All in all, this Agile stuff should be a Good Thing for you guys! Testing is no longer just an afterthought, it is a critical part of the project and everybody is involved with it all the time. Your expertise is finally being given the credit it deserves!".

Then I went on to talk about the history and principles of agile software development, and show how Scrum and XP works in practice. Here are the slides if you are interested.