What is the ROI of Testing?

In typical testing techniques and methodologies, testing is a rare resource. You should take into account risks that happen due to constant budgetary and time limitations, to get by until software that is being tested will have been finally released.

But if you have tools and time-tested approaches, that increase incomes from a testing process due to a high number of tests that are frequently updated, you should try to recoup a process of testing.

It’s impossible to test everything and even if you are sure that you have done this, you have obviously missed something.

You may use the following methods and tips:

1. Applying a notorious MoScoW priority when using every test case (Must Should Could Would — must be tested, should be tested, could be tested, would be tested). But sometimes it may happen that managers of a company think that only 20-30 % of created test cases must be tested and that the business simply needs numerous tests until it reaches the required test coverage. And so imbalance takes place;
2. The two-by-two combination and using equivalence classes allow decreasing the number of tests and focusing attention on low-level and informational fields, formats, not arduous business tasks;
3. Negotiations on if this test must be included in general regression, an integration set, or anything else. Sometimes regression tests are more important to a tester than testing a new (recently added) functionality. Frequently, integration and acceptance tests can be identical and so the question arises “What is the sense of testing everything for the second time?” Business needs not beautiful words but results that have a certain name.

Counting tests is…

A qualitative analogy to testing everything is a simple calculation of all possible decimal fractions. Each of them may hide one more inside and you can select any amount of numbers around for a random number.

Something similar happens to software tests. You can select any number of unique variations of tests (web environment, user’s behavior, time, preconditions, and so on). It’s very hard to estimate what can flow into each other since two tests can completely cover something general but be different in result.

Typical techniques mentioned above are simply techniques of a filtering process when initially an infinite sum of possible tests is simplified to something certain when every test is separated from another one. According to Aaron’s technology, it’s a “stone”. Then they are filtered and reduced to a certain sum that can be reached so we can give an answer to the question: “When will testing process be completed?”

Old costs of testing

The above-mentioned filtration is made due to the fact that every calculated test costs a certain amount of money and therefore, a test preparation process and a test run cost a certain amount of money.

Analytics found many years ago, that an average price of creating a typical test case can be reached in three hours and one hour is needed for a test run (maybe, with 45-50% probability of the second run or work on finding a bug).

In total, when there are 100 test cases, for example, a classical model of time and financial expenses will be the following: at least 450 hours, 3 hours for one test, with a mandatory second run in half of the cases.

Anyway, it’s not strange at all that managers want to decrease such costs and make costs as low as possible.

You should also keep in mind that in the best case, this may cover a test run only one time and 50% of them —  two times. Does this ensure complete safety to continue this work?

A fresh look at incomes

Most modern tools and methods of software testing turn this approach upside down and are focused on making a testing process as fast as possible, sometimes very cheap and not useful for further usage.

When ROI is calculated, a project will anyway comprise some percentage of automated tests: in other words, let’s say that only 75% of tests can be automated or can be maintained by the tools in such a way that they won’t stop during a test run.

For example, a test is run as a mandatory part of continuous CI/CD testing and you don’t need hands or eyes to do this.

Its preparatory work takes some time — for example, 3 hours again. Therefore, you need 112.5 hours to run 25 tests without automation but an automation process, that has a free test run, requires up to 225 hours of preparatory work. Even in this scheme of costs, they are 25-30 % lower, together with the second runs for the remaining 50% of tests.

The new approach is aimed at creating abundant and frequent tests, not the rarest ones. If we constantly use CI/CD and a whole team’s approach to quality, there will be positive dynamics with the highest usefulness for business tasks on the projects.

Conclusion

So to conclude the topic of a process of recoupment of software testing, you should remember the following:

1. A frequent test run is good since it provides maximum profit not only for a QA department but for all business processes;
2.  The performance of a company depends on the quality of executed tests: the better it’s established, the more profitable a business will be;
3. All test cases should be fast and be constantly repeated when choosing possible values.