Software Testing Services: Automated Functional Testing of Windows Applications by Means of Ranorex

With the advent of Windows clients in the software testing company, we faced a necessity to search for special technologies of automated functional testing for Windows applications, because automation tools for testing Web services (like Selenium) are not suitable in this case.

Search Criterion Were the Following:

• Development environment of functional testing services should support both auto recording of test cases (for testers with little experience in programming), and ability to record some parts of the tests manually.
• Since the client program is developed on C#, it would be desirable to write functional testing either on C# or on one of the .NET languages.
• Supporting of Data-Driven Testing.
• Using of XPath for UI elements identification.
• Code together with all the testing artifacts should be stored in the same version control system in which the primary code is kept (in our case – Git).
• The given technology of automated functional testing should be easily built into a continuous integration system (Jenkins).

After the preliminary market exploration, our testers stopped on 2 products:

1. HP QuickTest Professional. It is one of the most widely used products on the market though it is rather expensive.
2. Ranorex. It is a new product, but it is mentioned as a good alternative to HP QTP on professional forums.

Our testers checked and compared all the features, productivity, and usability of both products (Ranorex & HP QTP), examined the integration of Ranorex with CI Jenkins.

Ranorex Capabilities

Types of Applications That Can be Tested. The same apps that can be tested by HP QTP: Windows native (WinForms, WPF, Win32), Java, Qt, Delphi, Flex + HTML (Browsers – IE, Firefox, Chrome, Safari). Its last version also supports mobile client testing on Android and iOS.

Action Recording and Repositories of Items. Ranorex supports both, action recording by means of the built-in recorder (similar to HP QTP), and UI elements identification by means of Ranorex Spy.

All identified elements are stored in XML repository where each element is recorded by means of XPath notation. For example, a toolbar button can be described as:

Auto-generating of code in C# or VB.NET is performed based on the recording. Every step can be recorded manually as well (for example, repo.MyApp.MainPain.BtnOk.Click():).

UI Elements Identification. RanoreXPath makes the search of necessary elements easy. It supports the search of regular expressions (form[@title~’^MyApp.*’]), child objects (.//button), logical operators (@text=’OK’OR@controlname=’buttonOK’), built-in functions (table/row/cell/[first()=`True`]) etc. Every element in the repository has search-timeout. For example, if we work with a button on a window that (form[@title~’^MyApp`]) has search-timeout 30 seconds, in this case Ranorex is waiting for 30 seconds for the window to appear (for example, when the application is started), and then proceeds to work with the button. It allows refusing from the use of “wait” function in tests.

Repository Elements. We work with various types of elements in Ranorex (form, toolbar, container, tree, list etc.). As we work with elements indirectly but by means of element adapters, the element “button” will have a similar set of properties and methods in Windows native application, Java and Flex applications.

Integrated Development Environment. Ranorex Studio is based on free SharpDevelop. As a result, the organization of code and test cases resembles MS Visual Studio: all test cases are broken down into projects. Projects that are used together can be united in “solutions”. We get one EXE file for each solution and one DLL for each project at the output after compilation. It’s enough to copy them on a test machine and to start, like a usual Windows application.

Test Case Organization. Every solution has one test suit. We add the necessary test cases and drag the recordings from our projects in such a test. Recordings can be recorded automatically or manually in one of the supported programming languages. We can use any necessary code written in C# or VB.NET in manual modules, for example, an access code to the Windows registry, code for file downloading by FTP, etc.

Data-Driven Testing. Every test case can be connected with data from CSV/Excel file or from databases. Every module (record) can have variables connected with data or filled in the module.

Programming Language. C# or VB.NET are used for writing tests for Ranorex. VB Script is used in case of utilizing HP QTP. It is more difficult to learn VB.NET or C# than VB Script for people with no experience in programming. Scripting languages, as a rule, are more suitable for test automation. But some experienced testers prefer C#, a fully-featured object-oriented programming language with full support of .NET, instead of primitive VB Script scripting language.

Code Storing and Versioning. HP QTP partially stores code and repository in the binary format that makes its versioning by third-party means, like SVN or Git more complicated. Ranorex, Unlike HP QTP, stores code (including auto-generated code) and all the repositories in text format (for example, XML format is used for repositories). It allows using Git, with all its properties like local repository, tagging, etc. for versioning and co-developing.

Ranorex & Continuous Integration System

In contrast with HP QTP, Ranorex doesn’t have a system that allows starting tests on a schedule or an external trigger on remote PCs. On the one hand – it’s a minus, but on the other hand – we can integrate Ranorex tests with the Jenkins server without any problems with its help (any servers will be suitable, even TeamCity):

• Compilation of Ranorex Tests. There is Build_Ranorex_Tests job on Jenkins. Once one of the testers push the code in Git-repository, one of Git-hooks starts this job. It consists of two stages: the compilation of all the solutions (test suites), using MS Build (/t:Clean,Build /p:Configuration=Release) and then activation of all the files that are necessary for testing (EXE, DLL, CSV/Excel and others).
• Tests Performance. Then “matrix” job is launched on all test machines (we test for Windows XP, Vista and Windows 7) where Jenkins Slaves is installed. This job first installs the latest version of our application (using MSI), then runs sequentially EXE files that were compiled by the previous job. This results in one ZIP file for every test suite; each ZIP file contains 1 XML file with the outputs and a directory with screenshots.
• Results Presentation. After this script (it can be done as a separate suit in Ranorex that will run last) changes the format from Ranorex XML to xUnit format. Thanks to this there is a report in Ranorex format and in ф format suitable for Jenkins for every client. So, Jenkins can build the graphics of results and it mails Ranorex results to all the project members.