Results based testing

Results-Based Testing is a business model for software testing. This business model consists of an alternative pricing system which allows companies to pay for the bugs which are detected, instead of for time spent on a project.

Description

"Results-Based Testing" (RBT) is an alternative pricing system for software testing which allows companies to pay for the bugs which are detected, instead of for time spent on a project. This was adopted in response to dissatisfaction which clients expressed toward the pricing structure employed by most testing companies, and has led to higher customer satisfaction and better accuracy in bug detection.

Results Based Testing usually involves three elements:

1. A Scope of work
2. A contractual SLA
3. A pricing mechanism

RBT is normally utilized when part or all of the software testing process is outsourced to a third party and a core contractual SLA together with a pricing mechanism sets the exact payment made at each SLA level. The pricing mechanism may be a flexible rate for each SLA level or a Penalty/Reward mechanism, all with the goal of creating an incentive for the Testing Supplier to meet the business targets (results) set forth. However, RBT may (and should) also be used for internal testing teams even though in such cases a penalty/reward mechanism is harder to implement. Another good use of RBT is for establishing the necessary framework for measured continuous improvement in which results from previous periods can serve as a baseline for following period's targets.

Usage

Several software testing companies utilize this approach, including QualiTest, who rely heavily on the success that they have encountered through using this model.

QualiTest reports that Results Based Testing has provided benefits because of the following:

• Provide the testing provider with financial incentives to meet customer's business goals
• Provide the testing provider with financial incentives to innovate and improve the process in line with customer's business goals and spreads the financial risk between both parties
• Provides a framework for continuous improvement
• Measures Test Provider's performance.
• Provides the customer flexibility to scale the testing up or down in line with its business needs.

When evaluating the level of testing, several Key Process Indicators (KPIs) should be measured. The main focus should be on two main questions:

1. What percentage of the defects should be found by testing?
2. What is the cost spent in order to achieve the above goal?

Most organizations fail to measure these two KPIs and are unable to provide accurate visibility into the quality and efficiency of testing.

To measure the percentage of defects found by testing (a type of test Coverage KPI vs. the escaped defects KPI) the organization should utilize the following process:

1. Reporting of defects – each defect reported by the testing team should be documented in a central defect management system.
2. All issues or support tickets raised by customers/users of the system should be documented in a centralized system. Usually the support or help-desk team has this information.
3. Each ticket should be evaluated by the testing team (sometimes the support team filters the tickets and provides only the tickets that result from a defect).
4. Each ticket related to a defect should have one of the following statuses:

• Not a defect
• Known defect
• Cannot be found by testing/un-reproducible
• New Defect

Only defects in the last status (New Defect) are counted for this metric.

The process above is extremely important in case the organization is looking to start implementing RBT and to continually improve the efficiency and effectiveness of the testing process. Measuring the test coverage is done by dividing the amount of defects found by the testing provider with the amount of defects found by the users of the system. Since critical defects carry a different importance to the organization versus less severe defects, each defect is multiplied by its severity. For example, assuming a scale of 1-5 is used, a critical defect (severity = 5) will be counted in the same way as 5 minor defects (severity = 1).

Only defects found in a certain period after release of the system shall be counted (normally this is defined as 3 – 6 months). Once the data is available the following formula is used to calculate the KPI value:

(Σ defects found by testing)/(Σ defects found by testing+Σ real defects found by users)

References

• Zylberman, Ayal. "Results-Based Testing". www.qualitestgroup.com. QualiTest. Retrieved 17 July 2015.