Mobile Testing: Complete Guide

This article will be useful to everyone who is involved in the development process both on the part of the agency and the customer: managers, designers, developers, testers. It presents the main features of mobile application testing.

You can use one of the types listed below, depending on the aim you are pursuing:

• functional testing;
• performance testing;
• security testing;
• usability testing;
• compatibility testing;
• recovery testing.

Functional testing

Mobile application functional testing usually covers user interaction testing, as well as testing of transactions. Important factors for this type of testing are:

1. The type of application defined by its business functionality (banking, gaming industry, social networks, education).
2. The target audience (user, company, educational environment).
3. The distribution channel for the application (for example, App Store, Google Play or direct distribution).

In simple terms, we check whether the application performs the expected functions, which are usually described in the specification or dictated by the business processes.

Therefore, functional testing can be carried out in terms of requirements. In this case, test cases are formed. Technical specification based on the business processes is used for its creation. Once this is done, so-called use cases are created which describe scenarios for daily or permanent use of the application.

The basic functional testing scenarios:

1. Check the correctness of the mandatory fields.
2. Make sure the required fields are not displayed on the screen as optional.
3. Verify that the application operation during the running/exit meets the basic requirements.
4. Ensure the application goes into the background in the case of an incoming call. You will need one more phone to do this.
5. Check if the phone can store, receive and send messages while the application is running. To do this you will need another phone from which you can send a message to the tested device with the already running application.
6. Make sure the device works in multitask mode when necessary.
7. Check how the necessary options work with the social networks – Share, Publication, Navigation.
8. Ensure the application supports payment transactions through the payment systems Visa, Mastercard, Paypal, etc.
9. Verify the adequacy of the page scrolling scenarios.
10. Check the proper navigation presence between important application modules.
11. Make sure that the number of rounding errors is minimal.
12. Сheck for error messages, for example, «Network error. Please, try again later» in case of incorrect network operation.
13. Ensure the installed application does not prevent the normal operation of other applications and does not consume their memory.
14. Check if the application is able to return to the state in which it was before the suspension (for example, hard reset or system failure).
15. The program installation must take place without significant errors, provided that the device meets the system requirements.
16. Make sure the application automatic startup is correct.
17. Check how the application works on all the devices of the 2G, 3G and 4G generations.
18. Perform regression testing to identify new software errors in existing and already modified areas of the system. Additional conducting of all the previous tests to verify the program behavior after the changes.
19. Ensure the availability of user manual.

The system often possesses a lot of functions and it is not always possible to check them all. Thus, certain test cases and user cases are usually prioritized before the start of functional testing, regulate the time in accordance with the priorities and then pay attention to the most important ones. It is quite difficult to select some standard scenarios for functional tests due to the variety of applications but it is possible to identify frequently occurring modules, compose test cases for them and then to use it, modifying for specific requirements.

For each function, it is necessary to check both positive scenarios and negative ones. A scenario is considered positive when the user achieves a goal (create item, sends a message and so on). The negative, vice versa – an error occurs at some of the steps and the goal can not be achieved.
For example, consider login / logout and creating a contact (section, user, or any other item). Standard login / logout can include such options:

• registration: with a login and password, without a password, through the social network, etc.;
• authorization: with a login and password, through the social network, etc,;
• password recovery;
• logout: standalone, after the session, etc.

Positive scenarios:

• Registration in the application is available with all the methods described in the terms of a technical specification.
• You can register by filling in only the mandatory fields.
• You can register by filling in all the fields completely.
• It is possible to log in to the application after the registration. In this case, the entered data is correctly stored in the profile (e-mail, password, personal information, etc.).
• Registering on one device, you can log in to another – the data is available and correctly stored on the server and available.
• Logout works correctly.
• Password recovery works correctly.

Negative scenarios (the most obvious):

• Re-registration for the same e-mail with the same login is not available.
• Registration without filling in the mandatory fields is not available.
• Registration, if all the fields are left blank, is unavailable.
• Registration, if the format of the entered data does not meet the requirements, is not available.
• Authorization with empty fields is not available.
• Authorization with a wrong / deleted / blocked login is not available.
• Authorization with incorrect password is not available.

Creating a contact. It is logical to assume that if a user creates a contact, it should be possible to view, edit and delete it. This is the basic set of functions an item can possess.

Positive scenarios:

• Creating, editing, viewing and deleting of the contacts are available.
• Creating a contact with the minimum data set is available.
• Creating a contact with the maximum data set is available.
• When creating, all the data types, described in the technical specification, are processed correctly.
• Once created, a contact is available for viewing.
• The change takes into account mandatory fields / data / elements. Saving the contact without them is not available.
• Once deleted, the contact is no longer available.

Negative scenarios:

• Creating two identical contacts is not available (it can be a positive scenario).
• Creating a contact with a missing required elements / data is not available.

I can also add the user interface checking to the functional testing.

1. Screen verification for matching the layouts.
2. Checking the work of native gestures: swipe, multi-touch etc. – application should react to them in a certain way.
3. Checking the states of the elements: buttons change the color if pressed; lists are collapsed and unfolded, so on.
4. Verification of localization, if it is stated in the application. It is important to pay attention to layout. A lot of names in other languages are much longer than in English or Russian.

Performance testing

It is also known as stress testing. It is an automated testing that simulates the work of a certain number of users of a shared resource.

The main tasks:

1. Determine the number of users who can work with the application simultaneously.
2. Check how the application behaves when the intensity of any operations is increased.
3. Check the application performance at an average load for hours of use.
4. Verify the application behavior under stress conditions.
5. Check the work in an “overgrown” database – how quickly the requests are executed.

The main purpose of this type of testing is to make sure that the application is working at certain performance requirements: access to a large number of users, elimination of an important element of the infrastructure, such as a database server, etc.

The main scenarios for mobile application performance testing:

1. Determine whether the application operation is the same under different network conditions.
2. Find out if the current network coverage is able to ensure the application work at different levels of user load.
3. Determine whether the existing client-server configuration provides an optimal performance.
4. Find various application and infrastructure bottlenecks that reduce application performance.
5. Check whether the application response time meets the requirements.
6. Evaluate the ability of the product and / or hardware to cope with the planned load volumes.
7. Evaluate the time during which the battery can support the application in the conditions of planned load volumes.
8. Check the application in cases of transition from a Wi-Fi network to 2G / 3G mobile network and vice versa.
9. Verify that each of the processor memory levels works optimally.
10. Make sure that the battery consumption and memory leak do not go beyond the limits of the norm, and the operation of various resources and services, such as GPS navigation or the camera, meets the requirements.
11. Verify the application stability in conditions of a hard user load.
12. Check the network effectiveness in conditions when the device is in a motion.
13. Check the performance of the application if it works under conditions of a non-permanent connection to the Internet.

Security testing

It is a testing strategy used to test the security of the system, as well as to analyze the risks associated with providing a holistic approach to the application protection, hackers, viruses, unauthorized access to sensitive data.

The main purpose of this type of testing is to ensure the security of the network and application data.

The key steps to verify the security of your mobile application are given below.

1. Make sure that the data of users of the application, logins, passwords, bank card numbers, are protected from network attacks of automated systems and can not be found by selecting.
2. Ensure that the application does not give an access to a confident content or functionality without proper authentication.
3. Make sure that the application security system requires a robust password and does not allow the attacker to possess the passwords of other users.
4. Ensure the period for the session timeout is adequate for the application.
5. Find dynamic dependencies and take measures to protect these vulnerable sites from attackers.
6. Protect the application against the SQL injection attacks.
7. Find cases of native code and eliminate its consequences.
8. Make sure that the certificates have not expired, regardless of whether the application uses the Certificate Pinnig or not.
9. Protect the application and network from DoS Attacks.
10. Analyze the requirements for data storage and its verification.
11. Provide session management to protect information from unauthorized users.
12. Check all cryptographic codes and, if necessary, correct errors.
13. Make sure that the business logic of the application is protected and is not subject to external attacks.
14. Analyze the interaction of system files, identify and correct vulnerabilities.
15. Check the protocol handlers (for example, whether they do not try to reconfigure the landing page, using malicious floating frames, by default).
16. Protect the application from the malicious attacks on clients.
17. Protect the system from malicious implementations when the program is running.
18. Prevent possible malicious consequences of file caching.
19. Prevent unreliable data storage in the cache of the device’s keyboard.
20. Prevent possible malicious actions of cookies.
21. Provide regular monitoring of data security.
22. Examine user files and prevent their possible malicious influence.
23. Secure the system from cases of buffer overflow or memory corruption.
24. Do analysis of various data streams and protect the system from their possible malicious influence.

Usability testing

Usability testing is carried out to create fast and easy-to-use applications. The main goal is to ensure the usability of the application, to create an intuitive interface that conforms to accepted standards.

So, to conduct usability testing, you should:

1. Make sure that the buttons have normal size and are suitable for large fingers.
2. Put buttons in one area of the screen not to cause confusion for users.
3. Make sure the icons and pictures look natural in the application environment.
4. Ensure that the color of the buttons, which perform the same function, is the same.
5. Ensure correct work of the zoom-in and zoom-out system.
6. Provide minimal keyboard input.
7. Make sure that you can return or cancel the action if you press the wrong button.
8. Make sure that the context menus are not overloaded, since they imply a quick use.
9. Make sure that the text is simple, clear and visible to the user.
10. Ensure that short sentences and paragraphs are possible to read.
11. Find the optimal font size.
12. Make sure that the application warns about possible failures in his work when the user loads large amounts of information.
13. Ensure that the application can be terminated from any state and that it resumes its work in the same state.
14. Check that all the lines are displayed in the required language, if the application has a translation option.
15. Make sure the application components are synchronized with the user’s actions.
16. Provide the user with a guide that will help understand the application and use it effectively.

Usability testing is usually conducted with the help of users because only people can understand the subjective feelings, caused by an application, of other people.

Usability testing is a testing method aimed at establishing the degree of usability, learning, comprehensibility and attractiveness for users of the product under development in the context of given conditions.

Usability testing provides an assessment of the level of ease of  the application use for the following items:

• Performance, efficiency – how much time and steps the user will need to complete the main tasks of the application, for example, posting news, registering, purchasing (the less time and steps the user will need, the better it is).
• Accuracy – how many mistakes did the user make while working with the application?
• Recall – how long does the user remember how to use the application after suspending work with it for a long period of time? (Repeated execution of operations after the break should take place faster than for the new user).
• Emotional response – how does the user feel after the completion of the task: confused, stressed or he liked everything? Will the user recommend the system to his friends?

To improve usability, it is useful to follow these two principles:

1. “Protection from the fool”. If the field assumes entering a phone number, it is necessary to limit the input range only for digits and form the keyboard. The same is for e-mail and other elements that assume user input.
2. Use the Deming Cycle (planning-action-checking-adjustment). That is, to collect information about the design and usability from the existing users and plan changes in the application on the basis of their views.

Configuration testing

Configuration testing is done to ensure optimal application performance on different devices taking into account their size, screen resolution, version, hardware, etc.

The most important scenarios of configuration testing:

1. Make sure that the application interface matches the screen size of the device and the text does not go beyond the display.
2. Make sure that the text is easy to read on any device.
3. Make sure that the call / alarm function is available when the application is running, the application collapses or goes into standby mode in the case of an incoming call, and upon its completion it resumes.

It is a type of testing designed to test the application efficiency on various system configurations. It is worth to consider the client level of configuration testing. The server for mobile applications is often the only one and the client is installed on a large variety of devices. But you need to consider that if the application is specific, for example, a game where several servers are allocated, then the server layer also becomes a priority.

At the client level, we can distinguish:

• Device type: smartphone, tablet, etc.
• Device configuration: the amount of RAM, the type of processor, the screen resolution, the battery capacity, etc.
• Type and version of the operating system. iOS 6, 7; Android 4.2.2, etc.
• Network type: Wi-Fi, GSM.

It is recommended before configuration testing

• Create a traceability matrix (the traceability matrix is a table into which all possible configurations are entered).
• Prioritize configurations (in practice, most likely, all the desired configurations will not be verified).
• Step by step, in accordance with the prioritized priorities, check each configuration.

At the initial stage it becomes obvious: the more requirements for the application work with different configurations of workstations, the more tests we will need to conduct. In this regard, we recommend, if possible, to automate this process, because automation really helps to save time and resources during the configuration testing. Of course, automated testing is not a panacea, but in this case it will prove to be a very effective helper.

Recovery testing

Recovery testing checks the tested product in terms of the ability to resist and successfully recover from possible failures that have occurred due to software errors, hardware failures or connection problems. Mostly it is used in applications that have to work 24×7, where every minute of idle time is very expensive.

1. Verification of recovery after a system failure and a transaction failure.
2. Verification of the application effective recovery of the after the unforeseen crash scenarios.
3. Verification of the application ability to process transactions in conditions of a power failure (discharged battery / incorrect application shutdown).
4. Verification of the data recovery process after a break in the connection.

Other important areas of verification:

1. Installation testing (fast, adequate application installation).
2. Deletion testing (fast, adequate application deletion).
3. Network test cases (checking the adequate performance of the network under the different conditions of loading, as well as the network’s ability to ensure the functioning of all applications used during testing).
4. Checking the availability of non-functional keys.
5. Checking the application download screen.
6. Verify of the keyboard input ability to enter from the keyboard during network failures.
7. Checking the starting methods of the application.
8. Checking the availability of charging effect in case the application is in a background mode.
9. Check the economy mode and high performance mode functioning.
10. Identifying the consequences of removing the battery while the application is running.
11. Checking the application energy consumption level.
12. Checking the side effects of the application.

Separately, we will point out the integration testing, under which module interoperability checks are performed, as well as applications with the operating system and other applications.

This includes all scenarios related to various kinds of interrupts while working with the application:

• there was a phone call;
• it was received message;
• a notification of another application appeared on the screen;
• the network connection was lost or bad connection quality (gets here because this is a usual thing for mobile networks);
• the application was minimized to the tray;
• the application was put on the background mode etc.

These cases should be provided when developing and testing the application.