In the pursuit to automate testing to create faster feedback loops and build quality in, some teams forget the value of manual testing. In my experience, without manual testing (as well) we are toast.
It’s easy to underestimate the value of modern manual testing techniques. Automated tests (and any old-school manual tests following a test specification) will only find bugs where the test designer (or developer) has predicted them to be found. On a component or system level you will rarely have 100% automated test coverage, so there will always be “holes” where bugs can lurk.
Specified test cases (executed automatically or manually) work well for verifying that acceptance criteria are met (as far as we have been able to specify them) and protect against regressions. New bugs, however, are almost never found with automated tests. Instead, developers catch them before checking in by running the test cases in their own development environment. This is part of building quality in with tighter feedback loops.
Bugs in new software that have good test automation are found using modern manual test techniques such as context driven testing and exploratory testing. This is not random or ad-hoc testing. It’s a semi-structured approach that requires great skills and is designed to find bugs outside the test specifications and unlike automation takes into account how the system is actually going to be used. It’s a necessary complement to automated testing, especially on a system level and for acceptance testing.
Experienced testers at a bank (where they still had some manual test cases) tell me that they find 90% of the bugs doing exploratory testing and 10% in manual testing and 0% in automated testing. Their goal was to automate all test specifications so they can spend all their time doing exploratory testing in order to raise quality further.
In summary: To ensure the highest quality release, manual testing (of the right kind) is necessary.
Good points. It’s worth note that automated tests lack the ability to spot usability glitches. Imagination, is still a human trait.
In my mind, it is easier to think of automated testing as “checking”. That tells me that we still need to test. However, we have to accept that some testing will be done after release, IMO, as we else risk suffer from cost of delay.
A good reminder Jan of the different types of testing. I worked as a QA tester in a large insurance company many years ago and we found almost all of the bugs when we were creating the automated test with some slightly different data/options, not when we were running the automated tests.
I agree with Per that the automated tests as “checking”. I’ve used the concept as part of a “safety net” for developers, especially in larger codebases with spaghetti code or just lots of dependencies. At the end, automated tests should be used in conjunction with other types and not seen as the sole solution.
I agree.
Automation is for regression. Makes sure you don’t have to worry about the old stuff.
To find new bugs, you have to be creative.