Usage of Key Performance Indicator in Software Testing

Most likely, someone is already a professional in it, but for someone, such a question is still a big problem. The process of estimating the time for testing is completely a group task.

In addition, such an evaluation includes not only the stages of testing but also the initial estimation by a QA specialist, which later can undergo some global changes that were not previously reported to the client. Nowadays the KPI implementation in the field of testing is very relevant, and a lot of attention has been paid to this topic from specialized professionals.

KPI Concept

The KPI concept (Key performance indicator) is a kind of indicator of the reached success in a particular field of activity or in achieving preset goals. In other words, the KPI is a numerically measurable indicator of the actual results. In the case of testing, the KPI is an indicator of the QA department productivity within a particular quality assurance organization.

Why Do Testers Need KPI?

In fact, everything is very simple and clear: with the help of KPI, you can view the current state of the project and, on its basis, take preventive measures in order to avoid big problems (time delays, missing a large number of obvious errors, etc.). On the basis of KPI, any project manager can look through the strengths and weaknesses of the development, as well as track the weaknesses and strengths of his/her own management decisions during the product creation (which was the right decision, which solutions were more successful, and which were less) in order to be able to fix them in the future.

Moreover, you can add not only common numerical indicators to the KPI but also quality values (for example, “customer satisfaction level”).

Where Can We Get the Famous KPI?

Which metrics are the best to be used on a project? How to estimate them correctly? Is there any system in theory with a set of pattern metrics, comparing which you can get KPI?

Any project is a unique thing in many ways. You shouldn’t hope too much that metrics from one project will be perfect for another: they have to be created on the basis of each project specifics and potential expectations/concerns from the client.

Now we can consider how to estimate the time for testing based on real examples.

Here’s the Example

So, an imaginary testing team began testing software for a certain client. The product was initially considered as several functional components and also meant integration with back-office data storage systems (databases).

We note that by the “client” we will understand any person who is interested in the software testing in some way and wants to get the effect when the product can satisfy final users requests and go into production for multiple usages. The client contacted the testing company with his/her own expectations and a predetermined goal.

At the first stage, the project manager, or the senior test manager, should find out all these goals and expectations. In practice, there are many options for such an analysis: questionnaires, briefs, conversations. It is very important to find out as fully as possible what exactly interests the client, what he/she will have to worry about in the future, and what his/her “pain and suffering” are.

After the client’s potential expectations and concerns have been formed, the testing company is charged with the task of transforming them into the final goal. It is easy to guess that the aim of any test is complex for the integration and functional testing of client software.

Further, after discussing the purposes of the work, it is necessary to perform the decomposition procedure, in other words, to break the global functionality into the components, so that the involved QA team understands what exactly it is responsible for. Here, it would be best to describe in detail what exactly the famous decomposition is.

If the client wants to get the results of integration testing, then the QA team needs to perform the decomposition of the integration functional testing of the entire product. This will require analysis in order to understand exactly which parts the client system consists of, how many systems interact during data exchange in general, and in which systems future users will be able to perform certain actions.

In theory, everything is quite clear and simple: it is necessary to divide a large task into a number of small subtasks. Nothing is complicated, but in practice, it turns out that the QA department faces a trivial misunderstanding of the fundamental criteria for the task decomposition, and therefore further actions are done at random.

As a result, there are unequal load on testers, an incorrect estimate of future tasks, a misunderstanding of the given task and a logically incorrect idea of the work results. In order to understand this block better, you should refer to the principle of SMART.

The Concept of SMART and Its Principles

In fact, SMART is a special mnemonic abbreviation used by managers in various fields in order to memorize the principles of goal setting. Each letter of the abbreviation has its own technical decoding:

• Specific – definite. During the task developing, we have to understand from the very beginning what result we want to achieve;
• Measurable – can be measured. The project needs the tasks that can be measured (concerning time and labor efforts);
• Attainable – achievement. A smart project manager will never give an inexperienced employee a difficult task because the time spent on its solving will not be returned. Understanding the personal traits of each tester will allow a team to split the workload qualitatively, giving beginners simple tasks, and professionals – something more complicated;
• Relevant – essential. Is it possible to solve some task right now? And what if the team won’t have time? What to do then?
• Time-bound – time limit. Any task assigned to testers should be solved within a certain time limit. The setting of time bounds allows you to make the testing process as clear and controlled as possible.

As a result, we have a complete understanding of the fact of how a big task can be divided into several smaller ones. Now all the components should be analyzed.

They can be distinguished in the next set of actions:

1. We test all the functionality that was mentioned in the integration;
2. Create test data;
3. Verify the tasks for the next functionality upgrades;
4. Add the bugs to the bug tracker;
5. Test releases and hot-fix;
6. Check whether there is a possibility of transferring two or more priority products in each new software version.

Now it’s time to go through every subtask and analyze the metrics.

The List of Metrics Which Is Forming the KPI

Testing of the functionality given in the integration

How to perform these actions? You can, for example, use the proven methodology “% coverage of the XX software modules with tests”, which in the form of a classic Excel-table allows you to combine all the test groups into the one.

The Example of Excel-Table with Tests

Creating test data

You can use the metric “the number of modules / functional parts of the product for which 100% of the working entities were created.”

Verification of tasks for the next functionality upgrades

Here it is necessary to anticipate the potential client’s improvements and the average time that will be spent on the testing in the future. Such a solution allows you to determine quickly what exactly each version was aimed at (either bug fixing or functionality creating and implementing at the client’s request). As a result, you can easily analyze whether a team has time for the release of the planned functionality or not.

Add the bugs to the bug tracker

You can use several common metrics which can help you to get the most comprehensive information about the current state of the developed software version: “the sum of defects added to the bug tracker”, “the sum of the Blocker priority bugs on the software version”.

Test releases and hot-fix

In this case, the combination of such metrics is perfect: “% of test cases, created and executed on the project” + “% of the passed test cases’ success on the final version of the product.”

As a Result of the KPI Usage

Developing a KPI for any project is a very procedural and long-term task, but at the same time, it’s very useful and interesting. And we will tell you why.

After you integrate the necessary metrics for the project, you can get answers to such questions: what exactly the project team did excellently, in what indicators the testing level grew, if the management decisions and instructions were correct and effective. With the help of KPI, if a client calls, any test manager can easily answer the following questions:

• What is the current state of the metric;
• Which modules can be considered as the most problematic and critical;
• Which modules he/she needs to be careful with;
• What indicators can be set for priority versions;
• Can the product be released?

After using the metrics on any project, it will be so much easier to prepare progress reports for the client, and the entire QA project team will be able to make maximum efforts in the future so that the time for testing will be significantly reduced.

What to Do If the Project Was Released, but You Need to Re-Test It?

Traditionally, in this case, time is doubled, because the project team has forgotten everything, testers are busy with something else now, and therefore it takes a certain time to get to know the system better. Also, a non-standard cycle may be added, which means, that except the usual testing, it may need 100% regression – in this case, you have to add some time to the complete regression.

Conclusion

In conclusion, it can be noted that the easiest way to evaluate the time for testing is to start collecting effective metrics that in the future can be systematized and their most suitable groups can be used depending on the specifics of the ordered web product. Also, you can even borrow ready-made metrics for some projects from other companies in order to adapt them to your technical needs. The main thing is to divide correctly the project into sprints and modules, the analysis of which will allow the project team to note exactly how much time is needed for high-quality and effective testing.