CTS – Your Technology Partner

Testing Metrics – Requirements Pass Rate

Written by Craig Butler on December 6, 2012

By: Vicky Davis

The Requirements Pass Rate metric is one of several metrics that provide insight to the quality of a software solution. This metric indicates the status of test execution and the quality of the solution based on percentage of requirements passed. To obtain the values for the formula the tester must associate each requirement to a test. It may take several tests to cover a single requirement. If multiple tests cover one requirement, and a single test fails, the requirement fails. The values for the formula can easily be obtained from the requirements traceability matrix. The expectation for the Requirements Pass Rate at the end of testing is 100%.

The formula to obtain this metric is: Requirements Pass Rate = (Number of Passed Requirements Traced to a Test Case) / Total Number of Requirements

The following paragraphs provide guidance on interpreting the metric during a software development lifecycle.

Guidance During Testing
The Requirements Pass Rate should increase as System Testing progresses during system testing. If the Requirements Pass Rate does not increase but stays steady, then it may indicate that testing is taking longer than expected, blocking defects are preventing testing from progressing, or system testing is revealing a significant number of defects, which are slowing down testing. A steady pass rate could also indicate that defects are not being correctly resolved and as a result, the defect owners are reopening them. A Requirements Pass Rate of less than 100% as the Test phase begins to end could indicate that the quality of the solution may not be good, and a review of the open defects is necessary to determine the type and severity of defects uncovered and to take corrective action if needed by the review.

Guidance at Approval
The Requirements Pass Rate should be 100% at the approvals for moving into the Deploy Phase. Review the defects, tests, and test results to determine the impact to project if the Requirements Pass Rate is less than 100%.

Guidance After Deployment
If the Requirements Pass Rate is not 100%, then a review of the approvals is necessary to determine why/how the project team deployed the solution with a pass rate of less than 100%. It is possible that the project team decided to deploy the solution with a pass rate of less than 100%. This could be due to defects that the development team did not resolve but deferred to a future release or defects that the project team agreed not to resolve because the cost of fixing them was greater than their impact. The failed tests could also be associated with functionality that the team determined as low risk and did not test or pass due to their risk factor. If there are many deferred defects, it may be indicative of several things, such as the functionality was complex, and the development team could not implement the functionality during the time frame given for the project, or that the requirements were not clear enough for proper implementation. Reviewing the project, tests, and defects will help reveal the problem areas that caused the pass rate to be less than 100% and allow for process changes for future releases to achieve 100% pass rate.

Comments

comments