Mobile testing is more complex than web testing. While the latter concerns different browsers and a variety of versions, mobile testing focuses on Android and iOS versions and mobile screen resolutions. Besides stock Android, QA engineers have to deal with custom UIs. Most device manufacturers add them on top of standard Android to provide extra features and more distinctive design. Famous examples include OnePlus’ OxygenOS and Samsung’s One UI.
With this article, we’re aiming to make life easier by comparing dedicated mobile automation testing tools. We have included the recommendations of in-house and external experts who provide their opinion on mobile automation tooling.
How to test mobile applications
So, how does mobile testing happen? QA engineers use mobile automation testing tools to facilitate interactions of test codes with the subject application. The question is how to run mobile automation tests. There are three options.
Through live local mobile devices. With real USB-connected devices, QAs can test an app’s operability from the end user’s perspective. They can speed it up by running tests in parallel on several Android devices. Though it’s still quite time consuming due to the requirement of manual infrastructure settings, such testing provides the most accurate results. Alternatively, you can use cloud test services.
Using a real device cloud. To expand device coverage, QAs conduct tests in a remote testing environment - device farm aka device cloud. Preinstalled in OS and browsers, device farms provide testers with access to multi-versioned physical devices. Testers have a dashboard where they can manage all the connected devices. Sauce Labs and BrowserStack are widely used device clouds.
Via Android emulator / iOS simulator. QAs also test mobile apps via software that imitates a mobile device on their desktop. In particular, the Android emulator mimics both software and hardware environments of the target device, while the iOS simulator - only the mobile software. That’s why you need an original Apple computer for testing. Among the main pros of emulators, Alexander Kot, mobile automation engineer at AltexSoft, cites their ability to switch between devices very fast. Anastasiya Nalyhach, a test automation engineer at EPAM Systems, adds that emulators allow her to quickly launch local tests on a PC when there’s no real device available. However, emulators aren’t as reliable as real devices.
Android emulator in Android Studio
To understand how all these components make up the testing process, we first need to understand how mobile automation testing works on the whole.
How mobile automation testing works
To develop and test mobile devices the following IDEs are used: Android Studio and Apple Xcode. These IDEs have all the necessary instruments to work with code including code debuggers (e.g. Android Debug Bridge), emulators and simulators.
An interface that enables communication between a mobile device and your computer is called a mobile driver. The official Android drivers are UI Automator and Espresso. Developed by Google, both drivers are included in the well-documented Android Testing Support Library. Besides, there are third-party drivers like Robotium and Selendroid ( Selenium for Android). As for iOS, XCUITest driver ships with Xcode.
If you’re dealing with a mobile application available both on iOS and Android, writing separate tests for each OS will take a while. You can enable code reuse by taking advantage of a cross-platform tool. As a result, you write code only once and it’s compatible with both OS. The most popular example of a cross-platform mobile testing tool is Appium.
Three ways of using a mobile automation testing tool
Let’s sum up the whole process using Appium as an example. Once the code is developed, the Appium client sends commands to the Appium server via its API. In its turn, Appium converts the commands into the format accepted by the native test framework of the corresponding platform (UIAutomator, XCUITest) and, as a result, the initial commands are executed in the tested application inside a simulator or emulator. Alternatively, you may use a device cloud like Sauce Labs that has all the necessary devices pre-installed for you. So you just need to connect to the cloud platform. In case you decide to keep real devices, you'll have to connect and configure them yourself.
When to automate mobile testing
Often, mobile testing is performed manually due to the complexity of automating this process. Its touch screen actions like scrolling and pulling are quite tricky to automate. Let’s distinguish the cases when it’s reasonable to automate mobile tests and those when it’s wiser to stick to handheld checks.
Similar to web testing, there are multiple scenarios for automated app QA:
1. Long-term projects. Automation is worth it in the long run, if you plan to continuously develop and maintain the app.
2. Maintenance checks. If a mobile application is sourcing data from a remote server via an API, the app’s synchronization must be monitored on a daily basis. Often, changes take place on a server side leading to different crushes like login fails or failures to display some elements on the screen.
3. Smoke tests. If we’re dealing with a standalone app that doesn’t use external resources, we skip the previous step and proceed to smoke testing right away. Performed prior to release, it’s a superficial check of the basic functions.
4. Regression tests. It’s the most thorough check running all the routine tests that can be automated. Automation is especially cost effective for high volume regression.
5. Continuous integration pipelines. Since continuous integration revolves around automated tests only, there’s no other option here.
But don’t forget about manual testing. It will be more precise in terms of interruptions like incoming texts, calls, push notifications, etc. This also includes testing how the app performs depending on the different Internet connection type (3G, LTE, etc.) as well as offline.
So, it’s important to have both manual and automated mobile testing. While manual testing focuses on exploratory or usability testing, automation testing takes care of highly repeatable functional tests. For more details on manual vs automated testing, check our dedicated article.
Further, we overview the tools that will help you turn the by-hand testing of your app into an automated one. To learn about general automated testing tools, read our article on that topic. Here we concentrate on mobile testing, although some players will remain as they support both web and mobile testing.
Android and iOS open-source testing tools
Android mobile automation tools
There are multiple popular tools for writing automated tests for Android and emulating user behavior in the application. The first thing to check are the tools that Google itself has built for its operating system, UI Automator and Espresso. If you are used to Selenium in web environments, there’s a mobile version of it - Selendroid. You can also check popular and easy to use Robotium, which is supported by a large, open source community.
So, we’ll start by comparing two native Android testing tools - UI Automator and Espresso.
Espresso and UI Automator: a tandem for native Android testing
Initially designed for developers, Espresso requires access to the source code. So, it provides white-box testing in contrast to UI Automator. It supports Android applications starting with the 2.3 version. UI Automator was released later, so it maintains Android 4.3+.
There is a reason why Google keeps them both. Each framework has some gaps that the other one fills. While Espresso can operate only inside our test application context, UI Automator is able to check the interactions of the subject application with other apps. The frameworks are not interchangeable, but they complement each other. So, a better way is to use both frameworks for testing your mobile app.
Espresso and UI Automator pros:
Speed testing due to code synchronization (Espresso is faster)
Reliability: maintained by Google itself. If anything breaks, it’s fixed almost immediately
Built-in test recorder (only for Espresso)
An active community where developers are constantly contributing and replying to questions
Espresso and UI Automator cons:
Limited language support: Java and Kotlin.
Custom checks require good programming skills and a deep understanding of how Android works.
Both Selendroid and Robotium were developed prior to the official drivers. However, they are not as well supported as Google’s tools.
Selendroid: Selenium tool for Android compatibility testing
Selendroid is a mobile automation tool from Selenium, solely oriented to Android. But using ios-driver - a Selenium WebDriver API - along with Selendroid, you’ll be able to test iOS applications as well. Written on the Selenium Web Driver client API, Selendroid tests are fully integrated with current Selenium frameworks. Utilizing Selenium Inspector, a test case development tool, Selendroid monitors the current state of the app’s UI.
Selendroid pros:
The hot-plugging feature enables device unplugging without interrupting the test. Ultimately, it increases the number of devices that can be concurrently tested.
It has high compatibility with JSON Wire Protocol and Selenium 3 Ready.
Backward compatibility. For instance, it can support Android API 10 (version 2.3.3) through API 19 (version 4.4).
No app adjustments are required for automating it.
There is a built-in Android driver WebView app.
Selendroid cons:
There is no ability to automate the testing of activities outside of app like camera, maps, etc.
Slow: It’s hard to use on some machines with less than 4GB RAM.
Robotium: black-box UI testing library for Android applications
Robotium, one of the first automated testing tools for Android, is frequently utilized. As it’s used for black-box testing, there’s no requirement for additional data about the app’s structure or executed classes. The only thing Robotium needs to know is the name of the fundamental class and how to connect to it. Java is the preferred language for composing Robotium tests.
Robotium pros.
It permits to compose function, framework, and user acceptance test situations.
It has a large community: There are many open-source developers contributing to introduce more features in subsequent releases.
We asked Myhailo Skrypchuk, developer at Brandnic, about their use of Robotium to test a business name generator app: “We use the tool in operation testing to evaluate whether the operation has been completed by using the right data flow. Also, in security testing - we find and eliminate loopholes and vulnerabilities, launch different mock cyberattacks. In performance testing, we examine the app with memory requirements and its speed to complete the task.”
Robotium cons:
Tied to the one app process, akin to Espresso.
“If your application calls another one – Robotium can’t see it and perform any action there,'' adds Myhailo.
No ability to lock/unlock a smartphone or a tablet.
No record and play functions
Failure to provide screenshots.
iOS mobile automation testing tools
If you’re caught up in iPhone app testing, check out the options you have. It all comes down to XCUITest – Apple‘s official UI testing tool. That said, Google provides a nice alternative – EarlGrey, which you may find more suitable for your needs.
XCUITest: native methods to work with iOS UI
XCUITest runs within the Xcode environment and its tests must be written in Swift or Objective-C. Its version support starts with iOS 9.
XCUITest pros:
Distributed parallel test running
Built-in test recorder for simple tests, UI elements, and their characteristics
XCUITest cons:
Apple is slower than Google in terms of responding to bugs
Weak documentation: It doesn’t include a description of all the methods and how to use them.
Small community: You’ll find it hard to get an answer to your question.
Test running speeds are slow compared to Android tests.
Framework updates are rare.
Out-of-the-box testing only on simulators, requires additional utilities to work with real devices. One of them is Facebook WebDriverAgent (WDA). Running on a simulator or a real device, WDA transfers commands via API to XCUITest.
EarlGrey: Google’s answer to XCUITest
Developed by Google in 2016, EarlGrey is very similar to Espresso. Google itself uses the tool for functional testing of native iOS apps like YouTube, Google Calendar, Google Photos, Google Translate, etc. You can add Earlgrey to your Xcode project manually or using a dependecy manager .
As opposed to XCUITest, this UI automation framework shares the same process with the app: Test scripts wait for specific UI events to occur before directly starting an interaction with the application UI. Thus, the test can affect the shared memory changing the runtime behavior of the app. Although EarlGrey automatically synchronizes with the UI and network requests, it still allows for implementing customized timings.
Earl Grey advantages over XCUITest:
Brings Espresso’s synchronization features to iOS testing
Real devices support
Earl Grey cons:
Anonymous statistics reports back to Google
Only a single app testing
Cross-platform mobile automation testing tools
Mobile testing tools that work for both OSs will save you a lot of time and effort. While it’s easy to get lost in the pool of choices, we’ve selected the top five - both free and licensed - frameworks that can help you develop mobile cross-platform automated tests. Appium is the first thing that comes to mind. But it’s not so easy to figure it out, especially if you’re not on a first-name basis with Selenium. Sometimes, it’s better to pay extra for a script-free automator. We’ll review both options through the tools on this list.
Cross-platform mobile automation testing tools compared
Appium: a versatile mobile UI testing framework
If you’re skilled at Selenium, Appium will be a walk in the park for you. It’s basically a Selenium WebDriver designed for testing mobile devices. The core function of the Appium server is to receive test requests in a JSON format and convert them into commands that can be further accepted by the native testing frameworks.
How Appium server works
Appium- and WebDriver-based tests can run using WebDriverIO test automation framework.
It’s WebDriver bindings for Node.js. WDIO has its own test runner and is easily configurable. Its command-line interface comes with a nice configuration utility that helps create the config file in less than a minute. It also gives an overview of all available third-party packages like framework adaptations, reporters, and services, and can install them.
Appium pros. Appium is by far the most universal mobile UI testing framework.
It supports automation across a variety of platforms: iOS mobile, Android mobile, and Windows desktop.
The tool automates all kinds of applications: native, mobile web, and hybrid ones.
Appium is compatible with all popular programming languages: Ruby, Python, Java, JavaScript, PHP, C#, and RobotFramework. With clients written in each of these languages, there are no limits in terms of test runners and test frameworks. It’s actually inherent in Appium philosophy - place no restrictions on your test environment so that you can manage it the way you decide.
Unlike other automation tools, Appium provides a stable WebDriver-based interface keeping your tests easy despite any upgrades.
Example of Appium implementation for Android, Source: Thomas Hansson
Example of Appium implementation for iOS, Source: Thomas Hansson
Appium cons:
Anastasiya Nalyhach notes that Appium does not interact directly with an app, but uses HTTP protocol to send commands. Alexander Kot adds to that, “Commands execution takes a while due to a long interaction chain under Appium's hood: After launching a device and creating a session, Appium goes on to establish a connection with the WebDriverAgent server. Only then, the WebDriverAgent server communicates with the native Apple's test framework in order to execute commands on the device.”
Without access to the source code, there are issues with modifying the application state. In addition, Appium doesn’t allow for mocking the back-end part while verifying UI. In the end, be sure to have powerful hardware as Appium uses a lot of resources.
Calabash: Behavior Driven Development testing framework
If Xamarin is closer to your heart, go for Calabash. In 2017, Xamarin stopped supporting Calabash, and with the community’s help, Calabash turned into an open-source tool. There are libraries for each platform: Calabash iOS and Calabash Android.
Calabash ensures that an application is comprised of the right features, scenarios, and step definitions as well as whether it’s running with the right inputs.
Calabash pros:
Being BDD oriented, Calabash conducts development testing based on the application functioning.
As opposed to tool-agnostic Appium, Calabash is tightly coupled with behavior-driven Cucumber, using it to define test scenarios in simple language. This makes it comprehensible even for non-technical team members.
Calabash cons:
Calabash enables interactions with the app only for a Ruby-based test framework.
There are no recording and playback functions.
Test scripts debugging and test maintenance are the major issues. With the increase of the devices under test, it becomes difficult to maintain test data files.
Writing automation test scripts for Calabash requires a skilled and proficient tester.
SeeTest: commercial Continuous Testing platform
SeeTest by Experitest is the first commercial cross-platform testing tool on our list. Available with a 30-day trial period, SeeTest’s automate plan costs a user $199 per month if paid a year ahead or $249 each month. So why pay if there are free tools?
SeeTest pros:
Integrated Appium capabilities in a convenient interface with such features as device reflection, test recorder, object spy, and visual feedback
Expanded automation coverage including barcode & check scanning, audio features, GPS simulation and TouchID, or customized elements such as sliders, pickers, tables, gestures and any complex UI.
IDE plug-ins to develop and run tests on real devices directly from Eclipse or IntelliJ
Access to 1000+ mobile devices in the cloud and run hundreds of tests in parallel
Detailed test reports with screenshots, videos and log files.
SeeTest Reporter
To sum up, SeeTest perfectly fits into the CI process as it allows for prioritizing, scheduling and distributing multiple tests in parallel on the devices and OSs relevant to the specific build.
SeeTest cons. There is always a but. In the case of SeeTest, it is its price so the tool isn’t suitable for small scale projects. In addition, the tool isn’t open source so there’s no room for adjustments.
TestComplete: a proprietary script-free mobile testing module
TestComplete is a product of SmartBear, a software test automation leader. It offers automated modules for mobile, desktop, and web testing. With TestComplete, you can create tests by recording them or editing test commands in its panels and editors. You either run tests within the platform or export them to run on external applications.
TestComplete pros:
There is a wide range of programming languages (e.g. JavaScript, Python, Jscript.)
Script-free record and replay tool allows for recording multi-touch gestures (swipe, pinch, drag, drop, or scroll) and playing them back. Steve Pritchard, IT Consultant for Anglo Liners, claims that such tests are as thorough as manual ones.
TestComplete playback of touch-screen gestures
Specific checkpoints verify the application state during the test run.
Automatic checks validate key aspects. Steve confirms it: “Thanks to the real-time information the tool produces, we’re able to track the progress and status of our pages from any device simultaneously.”
There is pre-integration with SoapUI API testing tool.
TestComplete cons:
This solution is pricey. For a mobile module along with the platform itself, TestComplete will bill you $2,299 per year at a minimum. That's only for a single computing device. Such extensions as parallel test execution, video test replays, AI-powered visual recognition, automatic web audits for performance, accessibility, and SEO have extra price tags. Also, TestComplete charges separately for training and certification. With a purchase, TestComplete clients get access to the Case Management Portal where they can turn to for support. Before making any commitments you can try it out for a month.
Tricentis Tosca: licensed model-based Mobile Engine 3.0
As a part of the Tricentis Tosca standard installation, the Tosca Mobile Engine 3.0 has to be explicitly selected via the custom setup. Paired with Appium, Tricentis lets you leverage Appium’s power without having to deal with its technical complexity. Besides, the Tosca Mobile Engine 3.0 uses components of AndroidSDK and Xcode tools. It comes integrated into device clouds: Perfecto, Mobile Labs, Sauce Labs, and TestObject. This way you don’t depend on test environments but can engage them when the need arises.
Tricentis Tosca pros. Similar to TestComplete, Tricentis enables scriptless test automation along with manual code writing which makes it easy to use. Among Tricentis capabilities you can benefit from are:
Synthetic test data generation,
Risk-based testing,
Test case design,
Distributed execution,
Service virtualization,
Test management and analytics,
DevOps and Application Life Cycle Management tool integrations
Automated deploy and startup with tests launched automatically and no manual interaction required.
Tricentis Tosca cons: Tosca has custom pricing which is on the high side. Make direct contact to find it out for yourself.
Challenges of mobile testing (as compared to web testing)
In terms of functionality deployment, mobile development relies on the existing web feature set. So, creating a supporting app should be easier. But this doesn’t relate to the testing process. Here, mobile testing starts from the outset no matter how thorough the web testing was conducted. Now, we’ll overview the main difficulties mobile testing has in contrast to web testing.
Device fragmentation. As in the case of web testing, where we’re dealing only with PCs and laptops, mobile testing covers phones, tablets, smartwatches, etc. So, handling many hardware manufacturers with various screen sizes and custom UIs, complicates the testing process.
Double work. Web testing deals only with one environment - a browser where multi-versioning isn’t as big a deal as it is for mobile testing. Keeping an eye on both Android and iOS mobile operating systems, testers must find two types of page object locators and implement separate tests for similar features. Supporting two sets of infrastructure at a time, QA must be ready for any push updates that can cause test breaks.
Matching devices. As always, iOS requires an Apple computer for writing automated tests.
Testing speed. Web testing is performed directly in the browser. This is in contrast with mobile testing where the code goes through a longer chain of actions before it can be executed on the target device. This results in delays in the mobile testing process.
Tooling and support. Unlike web automation, mobile has a less mature community unable to address some questions. Its tooling is still evolving.
