CTS – Your Technology Partner

Manual Testing: Why and When?

Written by Jeff Lether on March 9, 2016

Start-up Team

Many software developers instinctively know they need to test their solutions, but they aren’t always clear on the best way to proceed. There’s manual testing versus automated testing. There’s scripted testing versus exploratory testing. There are many other choices and possibilities to consider. Navigating the variety of choices to pick the approach that best fits the situation can be challenging. One path many solution developers choose is manual testing, often because it’s an approachable option that’s easy to get started. To ensure that manual testing will be a good fit for your project, it’s important to understand the goals you hope to achieve as well as what kinds of projects are good fits for manual testing. This makes sure that manual testing will work well for you, and that you execute your testing effort appropriately to meet your objectives.

Why – What goals should you pursue?

Manual testing is above all, testing, and so it shares some common goals with any testing effort along with other goals, specifically related to manual testing.  Some overall testing goals you’ll want to ensure your manual testing achieves are:

  • Accurately measure quality – This is job #1 for any testing effort. If the results don’t tell you whether your solution is correct or not, it’s difficult to say if you have received value from your testing. The key point that’s often missed is knowing what determines correct behavior to begin with. There are different options to determine correct system behavior, such as: formal requirements specifications, training materials, user manuals, or other forms of documentation. More important than which ones you use, is to choose the most detailed functional description available, and use it consistently to decide what tests you should run and how to determine if they passed.
  • Provide traceability and coverage – You’ll want to connect your tests back to the source used to identify them. This way you can tell when you have enough tests to completely check the entire solution, or make an intentional decision to leave some less-important areas without tests. With this connection back to functional documentation in place, you can also determine what proportion of the system you have tested by executing a certain set of tests. Additionally, when you run a test that fails and create a defect to track the error, you’ll want to connect the defect to the test that produced it. This way, you can tell what test needs to be repeated to validate the fix for the defect, as well as identify potentially unstable areas of the system that may need further testing by observing what functional portions of the system tend to have high numbers of defects.
  • Be repeatable and reliable – It’s likely you’ll need to execute your manual tests more than once, either after defects have been identified and corrected, or for future releases of the solution. It’s important to be able to execute the test the same way each time to ensure a consistent measurement of solution quality. Otherwise, variations in how a test is performed could lead you to conclude a defect has been corrected when it hasn’t, or that functionality was broken by a recent change when it wasn’t. The key to making your testing repeatable and reliable is to document written test cases with a formal structure for your main body of tests. By writing down the test in detail, including such information as step-by-step instructions to execute the test, what test data to use, what preparation is needed before testing, and exactly what result you expect for each step, you’ll ensure that even when tests are run repeatedly by different testers at different times, you get consistent results that support an apples-to-apples evaluation of solution quality. An additional key to repeatable testing is to include steps to reproduce in any defects you create when testing. This allows the tester to reliably validate that the defect has been fixed when re-executing the test.

Some goals more specific to manual testing to keep in mind are:

  • Gauge user experience – Automated testing has many benefits that we’ll explore in other blog entries, but a unique strength of manual testing is evaluating the user experience of using a software solution. Since user experience is defined as how easy and intuitive a software solution is for a human to use, a human tester will naturally be much better at evaluating this than an automated test script. Automated scripts won’t be able to tell you that a given process doesn’t flow well, or isn’t readily understood by a user, or is visually unappealing. For solutions where user experience is an important component, manual testing provides a unique ability to verify that the user experience is on target.
  • Try unexpected scenarios – While automated testing can accelerate the use of detailed formal test cases, there’s also value in attempting unplanned test scenarios to see how the solution responds to unexpected user actions or inputs. This kind of “off the map” testing is called exploratory testing, and human manual testers are particularly good at it. Humans possess a unique ability to observe how a test is proceeding, note that something doesn’t feel quite right, and ask “What would happen if I tried to do this?” This can be an important supplement, not a replacement, for the documented test cases previously identified.

When – What projects are best suited for manual testing?

There are numerous factors a team considers when choosing the best testing approach. While no approach works optimally in all circumstances and there are often tradeoffs in any situation, there are clear markers that indicate a project may be a good fit for manual testing. Here are some to use in evaluating your project:

  • Is user experience an important consideration?
  • Is the solution new and undergoing significant change to functionality and user interface?
  • Is the solution one of your less critical applications?
  • Is the solution infrequently updated?
  • Is the functional path through the system highly variable?

If the answer to any of these questions is “yes”, then manual testing may be a good fit for your project!