Context Appropriate Performance Testing at Let’s Test

The Let’s Test conference (“A European conference on context-driven testing – for testers, by testers”) kicked off today in Sweden. I know, I’m not a tester, so why was I at the conference? Certainly it wasn’t the “context-driven” that drew me, since before I heard of the conference I didn’t really know what that was. The “for testers, by testers” wasn’t so inviting to me as a developer, even though as a member of an agile team I write unit and regression tests, and participate in functional and exploratory testing.

It was actually Scott Barber’s tutorial description that got me to sign up:

Context Appropriate Performance Testing, from Simple to Rocket Science

When most people think of performance testing, they think about the hard parts – the very hard parts. They think about the expensive and complicated tools that are required to simulate the activity of thousands of end-users all at the same time, while collecting tens or hundreds of thousands of measurements.

In reality, many performance issues can be detected and diagnosed with exactly the tools and knowledge you already have at your disposal using information obtained from quick, easy and cheap performance tests

I’ve worked at several different companies over the years, and performance testing is always one of the more difficult parts that is usually left to the end of the project. Performance testing steps into the lime-light when production crashes, or customers complain. Sometimes developers start worrying once all the functional requirements have been met and the product is ready to ship: “will the system stand up to real users”? Most often we think of performance too late, and as Scott said the solutions we think of the the “hard parts”: We need to reproduce the production environment exactly, we need to reproduce the exact load, we need to reproduce the system interactions exactly, and we need a tool to help us sift through all the data. Well, that’s pretty insurmountable a week away from release, or when production has already crashed.

I really wanted to learn how to get from “it’s impossible to performance test” to “we’re performance testing!” Even though the tutorial description promised that I would learn how to do that, I wasn’t too confident that it could be achieved in one day. I’m glad to say I was wrong! What a great tutorial. You can find the entire presentation here, but the biggest takeaway for me was the “Simple items”:

  1. Make performance a priority
  2. Give performance visibility
  3. Ask lots of questions
  4. Research the competition

The bottom line is that talking about performance, planning for it from the beginning, asking questions, and having a bit of data gets you very far. I especially like the breakdown presented in “Give performance visibility”, where Scott discusses what to performance test and when. It also turns out that there are a lot of free tools available for performance testing. I already knew about tools such as Apache Bench, YSlow and JMeter, but there’s a world of resources out there, you just need to know how to interpret the numbers 🙂

I would have been really happy with the tutorial if it had ended right there, but the last exercise was really the icing on the cake. We played the game “Set”. The deck of cards has four features: color, shape, fill and number. The cards in the set have to all have the same feature or all be different for each feature. We had to find as many sets as we could and organize them in a way where you could quickly see how many sets there were. The point was that even with just four features or constraints it’s pretty difficult to keep track of all the things that were similar even in a small controlled game, let alone when you’re analyzing massive amounts of performance data. So look for the four most important things and start there, for example: response time, load, cpu, memory!

I’m really glad I got the chance to participate in this conference, it’s too bad that we get so stuck in our roles that we don’t often go to a conference that doesn’t “sound” like it would be a good fit. It was great to be at a testing conference as a developer, even though I didn’t always follow the discussion, especially a question at the keynote that ended up touching on “ISTQB”, “SBMT”, “rapid testing” and their relation to “context-driven” testing!! Luckily, with a smart phone, and helpful lunch company you can get a lot of answers 🙂

Get in touch via my homepage if you have questions or comments!