According to a recent Capgemini survey, 48 percent of QA teams admit to relying heavily on manual testing. While technology doesn’t yet allow us to stop manual testing altogether, it’s still an interesting statistic. Test automation exists to reduce the big-project testing process that can’t be possibly done by a human. It will notice details that humans can miss, especially on repeated tests. Yes, there are elements of software, especially its interface, that machines can’t yet properly evaluate, but when it comes to bug detection and results recording, automation holds all the aces.
Any long-term engineering entails using automated testing. When it comes to performance and repeated testing, you can’t do without automation. But then why do so many development companies still rely on manual testing?
Cost. If you’re not using open source tools, it's hard to justify testing automation benefits. A lot of organizations simply lack the budget to acquire new software, train their employees or hire more staff, and update hardware for successful automation. It’s hard to measure ROI when you’re working on short-term goals.
Time. Using automated testing is basically writing code to test other code. Add to that the time spent setting up and maintaining the framework, developing new skills, and the road to automation has costly tolls. Sometimes it will result in having to write code more than to actually test.
Lack of expertise. A lot of smaller organizations don’t or think they don’t have enough expertise to unleash all possibilities of testing tools. And that’s without considering all those who don’t care about thorough testing overall and prefer focusing on releasing the product. Even if that’s the case, you will benefit from exploring solutions, specifically designed to help you test quickly and effortlessly for no cost.
For a full overview of both approaches and when to apply each of them, check out our article on manual vs automated testing. If you’ve considered all drawbacks and decided that you want to invest in automated testing tools to gain all the benefits, take a look at our test automation tools comparison table before diving deeper into each framework.
Manual testers work by going through a program just like a regular user, tediously exploring all screens, trying different inputs and actions to finally detect failures and record observations. While you still need humans to detect usability problems, a machine can replace a lot of long repetitive tasks with a click of a button. But first, developers must write test cases.
While manual testers are not required to be skilled at coding, automated testing traditionally assumes that testing scripts are written by developers. This approach requires that testers know how to code to some extent. In an Agile environment, testers and developers are not separated in different departments and their skillsets are mingled.
However, today codeless testing software is on the rise. Some of the tools on the market offer testing solutions with no coding required, which allow any member of the team to perform tests. This is made possible by a tool’s ability to translate written text into the programming language, hiding code under the layer of user-friendly UI. While this may be a great solution for an organization with a limited number of developers, it’s still a pretty restricted option. Without access to code, you lose on customization and detailed feedback that’s kept in the core. It’s each company’s own decision: to opt for simple and accessible solutions or enable their employees to learn the basics of some simple language like Python.
As we’ve defined what automated testing is all about, let’s take a look at available solutions.
Simultaneous tests. Selenium allows you to run tests on different browsers using different machines at the same time. So, you significantly reduce the testing time, which is especially valuable for big projects.
Mobile support. Using Appium, an open-source tool with the same API as Selenium WebDriver, you can utilize your Selenium skills to test mobile web, hybrid, or native apps. In case you're involved in mobile testing, we have a full-featured article about the dedicated tools, Appium included.
No image verification. There’s no possibility to check if images are displayed or loaded correctly. You’ll have to either use another library or do it manually.
Depending on the desired modules (mobile, web, or desktop), you can customize your pricing by choosing only desktop or only mobile testing capabilities. The license is a one-time purchase, though you’ll have to pay for maintenance annually. It retails cheaper than many other tools but it can still be pricey for starting businesses. There’s a free 30-day trial if you want to try it before you buy it. You can also buy certification or online training.
Customization. Besides using the visual interface, you are allowed to write or edit script manually if you find the editor tool insufficient.
Timely updates. As this is a commercial product, you can expect a high level of maintenance, customer support, and of course, updates. The documentation is also complete and you won’t have to browse through forums looking for a tip or a solution.
Support of desktop apps. While Selenium can only execute tests in a browser (and mobile apps using Appium), TestComplete supports Windows applications.
The sales team will help you with custom pricing though the reviews say it’s on the high side. Considering the breadth of Tosca’s capabilities, the cost may be justifiable, but you can always start with a trial to be sure.
Smooth learning curve. As the tool was intentionally created to be used by non-developers, it’s easy to set up and learn, allowing you to start automation tests right away.
At first glance, Katalon Studio has everything other tools lack. It’s completely free, supports web and mobile testing, runs on both Windows and Mac OS, has an active community, and a beginner-friendly UI. Let’s unpack this much-loved software.
Unified bundle. Setting up is pretty easy as you only have to download additional software if you test on mobile (Appium + XCode/Node.js).
Abundance of tutorials. Katalon has a well-organized library of tutorials with images and videos to help you out. Besides the official help, you will find lots of free courses and YouTube lessons from other testers online.
Visualized reports. Katalon automatically transforms your testing results in handy graphs to allow you to clearly see how execution goes.
Small (although growing) community. Even though its knowledge base is extensive, it can’t compete with Selenium in number of users. Which means you’ll have a hard time finding updated articles and reviews.
Collaboration capabilities. UFT has a shared repository where members of you team can store artifacts, functions, and spreadsheets.
Price. The tool is one of the priciest on the list, considering it doesn’t allow for testing desktop apps like TestComplete does. Despite being less than reasonable, it can be a huge stepping stone for many developers. A year's seat-based license will cost you $2,500 and a concurrent one - $3,500. UFT doesn’t offer any group discounts though they may provide one in their custom enterprise package.
Only Windows support. Just like several other testing tools that we’ve featured, UFT can’t be used on a Mac without additional software.
Small community. There are not many complaints about the software online but there’s not enough written overall. Not only is it hard to find fellow developers, the documentation and contributed content is limited to what Watir lists on its website.
Seamless integration with CI tools. To successfully use Ranorex in CI development, you can easily integrate it with Jira, Jenkins, Bamboo, or TeamCity, the most common continuous integration services.
Complex GUI testing. Visual interface testing is a complicated task usually delegated to manual testers. Ranorex automates it by using object recognition and following different user scenarios while freeing testing staff from repetitive tasks.
Price. It’s a fairly inexpensive tool for the amount of functionality it offers. You can grab it’s one user license for $2,800 and add additional machines and users for an extra $850.
If you're interested in a more detailed description, read our article focused solely on Ranorex.
Let’s quickly recap the key points you and your team should consider when choosing an automation tool:
Platforms and languages. As you can see, not all options offer a wide range of scripting languages to choose from, so it may be a quick no from you to the tools that don’t offer your preferred technology. The market is especially small for Mac users, so you may want to consider getting a virtualization program.
Coding skills. You definitely want some flexibility when it comes to your testing staff. Even if you’re not yet planning to invite less skilled testers to your team, the time may come when you want your expert developers to work on more demanding projects. Select a solution that provides this freedom.
Continuous Integration. CI is becoming a norm in the development and testing world so it’s better to focus on tools that were built to work smoothly in an Agile environment. See which of your preferred scheduling tools they support and what other capabilities your team can make use of.
Any long-term engineering entails using automated testing. When it comes to performance and repeated testing, you can’t do without automation. But then why do so many development companies still rely on manual testing?
Cost. If you’re not using open source tools, it's hard to justify testing automation benefits. A lot of organizations simply lack the budget to acquire new software, train their employees or hire more staff, and update hardware for successful automation. It’s hard to measure ROI when you’re working on short-term goals.
Time. Using automated testing is basically writing code to test other code. Add to that the time spent setting up and maintaining the framework, developing new skills, and the road to automation has costly tolls. Sometimes it will result in having to write code more than to actually test.
Lack of expertise. A lot of smaller organizations don’t or think they don’t have enough expertise to unleash all possibilities of testing tools. And that’s without considering all those who don’t care about thorough testing overall and prefer focusing on releasing the product. Even if that’s the case, you will benefit from exploring solutions, specifically designed to help you test quickly and effortlessly for no cost.
For a full overview of both approaches and when to apply each of them, check out our article on manual vs automated testing. If you’ve considered all drawbacks and decided that you want to invest in automated testing tools to gain all the benefits, take a look at our test automation tools comparison table before diving deeper into each framework.
Manual testers work by going through a program just like a regular user, tediously exploring all screens, trying different inputs and actions to finally detect failures and record observations. While you still need humans to detect usability problems, a machine can replace a lot of long repetitive tasks with a click of a button. But first, developers must write test cases.While manual testers are not required to be skilled at coding, automated testing traditionally assumes that testing scripts are written by developers. This approach requires that testers know how to code to some extent. In an Agile environment, testers and developers are not separated in different departments and their skillsets are mingled.
However, today codeless testing software is on the rise. Some of the tools on the market offer testing solutions with no coding required, which allow any member of the team to perform tests. This is made possible by a tool’s ability to translate written text into the programming language, hiding code under the layer of user-friendly UI. While this may be a great solution for an organization with a limited number of developers, it’s still a pretty restricted option. Without access to code, you lose on customization and detailed feedback that’s kept in the core. It’s each company’s own decision: to opt for simple and accessible solutions or enable their employees to learn the basics of some simple language like Python.
As we’ve defined what automated testing is all about, let’s take a look at available solutions.
Selenium WebDriver
We won’t even try to be original and we'll start with probably the most classic automation testing choice of all - Selenium WebDriver. It’s open source (meaning free), supports multiple programming languages, including Java, Python, C#, PHP, JavaScript, Ruby, and Perl, and runs on all three main desktop platforms: Windows, Mac, and Linux. It’s a robust testing tool that directly communicates with a browser to automate. The tool’s team also offers a limited but useful for prototyping plug-in - Selenium IDE, which records your actions and uses them to test accordingly in the future. To learn more about the product, read our full Selenium review.Why it’s good: the pros of Selenium WebDriver
Large community. Since its creation in 2004, the tool has acquired massive support. The community is active, rich with numerous blogs, forums, and discussion boards to help find a solution to your problem.Simultaneous tests. Selenium allows you to run tests on different browsers using different machines at the same time. So, you significantly reduce the testing time, which is especially valuable for big projects.
Mobile support. Using Appium, an open-source tool with the same API as Selenium WebDriver, you can utilize your Selenium skills to test mobile web, hybrid, or native apps. In case you're involved in mobile testing, we have a full-featured article about the dedicated tools, Appium included.
Where’s the catch: the cons of Selenium WebDriver
Not beginner-friendly. Writing scripts on Selenium WebDriver requires a technically capable engineer meaning you can’t entrust automated testing to less skilled people in your team.No image verification. There’s no possibility to check if images are displayed or loaded correctly. You’ll have to either use another library or do it manually.
When does Selenium succeed?
Regardless of its numerous competitors on the market, Selenium is still considered a standard and a solid choice for any testing project. Plus, if you’re looking for a prototyping tool requiring minimum programming knowledge, you may make use of Selenium IDE.TestComplete
TestComplete is one of the code-less tools that we spoke about before. It’s a proprietary software that can automate testing for web, desktop, and mobile applications. TestComplete is used for data-driven, keyword, unit, functional testing, and many more, making it a comprehensive software for daily testing. It supports a wide range of languages and frameworks including VB, JavaScript, JScript, C++, C#, Delphi, Angular, Ruby on Rails, PHP, and others.Depending on the desired modules (mobile, web, or desktop), you can customize your pricing by choosing only desktop or only mobile testing capabilities. The license is a one-time purchase, though you’ll have to pay for maintenance annually. It retails cheaper than many other tools but it can still be pricey for starting businesses. There’s a free 30-day trial if you want to try it before you buy it. You can also buy certification or online training.
Why it’s good: the pros of TestComplete
Ease of use. A built-in editor allows users of any programming knowledge to add and delete tests, modify parameters, and change test order.Customization. Besides using the visual interface, you are allowed to write or edit script manually if you find the editor tool insufficient.
Timely updates. As this is a commercial product, you can expect a high level of maintenance, customer support, and of course, updates. The documentation is also complete and you won’t have to browse through forums looking for a tip or a solution.
Support of desktop apps. While Selenium can only execute tests in a browser (and mobile apps using Appium), TestComplete supports Windows applications.
Where’s the catch: the cons of TestComplete
No Mac support. TestComplete doesn’t allow for Mac applications testing. If you want to test iOS apps on a Mac computer, prepare to acquire some virtualization software. Though you can still test both iOS and Android apps on Windows, which applies to all frameworks we’re describing today.When does TestComplete succeed?
Regardless of its benefits, the question of comparing TestComplete vs Selenium will always stand. If the budget allows, TestComplete can become a great tool for a team of less skilled developers. By buying modules on demand, you can customize the pricing and leverage excellent technical support. You can purchase a single license for about $4,600 and concurrent licenses for $9,000. To customize your order, you can buy each module at $2,600 apiece.Tricentis Tosca
Tosca by technology company Tricentis was created with Agile and DevOps in mind. It is well-known to be an exhaustive tool for all testing needs, but it’s still easy to use it together with any testing software the company is already using. Just like TestComplete, Tosca supports mobile, web, and desktop (only Windows; Mac and Linux with virtualization tools), doesn’t require writing script but manual code writing is still possible.The sales team will help you with custom pricing though the reviews say it’s on the high side. Considering the breadth of Tosca’s capabilities, the cost may be justifiable, but you can always start with a trial to be sure.
Why it’s good: the pros of Tricentis Tosca
Great for Continuous Integration. Tosca fits perfectly in the Continuous Integration workflow. This is a modern approach capable of dramatically decreasing development time and running numerous tests a day. Using scheduling tools, Tosca can run test cases directly from them and then transfer the results back as an xml file.Smooth learning curve. As the tool was intentionally created to be used by non-developers, it’s easy to set up and learn, allowing you to start automation tests right away.
Where’s the catch: the cons of Tricentis Tosca
Limited knowledge base. You won’t find a lot of community support as Tricentis’ own Knowledge Base and Forum are the only active places to leave your message. Although the documentation is easy to navigate, you may find yourself restricted after dealing with much more popular solutions like Selenium.When does Tosca succeed?
Tosca is perfect for Agile development with integration of various CI technology available. It’s a comprehensive testing tool with a low entrance point used by Accenture, Capgemini, Deutsche Bank, and many others.Katalon Studio
Katalon, built on Selenium WebDriver, is considered Selenium's logical and improved follow-on. It provides templates and common actions for keyword-driven testing, can be integrated with multiple tools for Continuous Integration, and has a rich, analytics-filled dashboard.At first glance, Katalon Studio has everything other tools lack. It’s completely free, supports web and mobile testing, runs on both Windows and Mac OS, has an active community, and a beginner-friendly UI. Let’s unpack this much-loved software.
Why it’s good: the pros of Katalon
Good for both pros and non-techies. Katalon hides all the complexities behind its interface but still allows skilled programmers to reach the scripting mode.Unified bundle. Setting up is pretty easy as you only have to download additional software if you test on mobile (Appium + XCode/Node.js).
Abundance of tutorials. Katalon has a well-organized library of tutorials with images and videos to help you out. Besides the official help, you will find lots of free courses and YouTube lessons from other testers online.
Visualized reports. Katalon automatically transforms your testing results in handy graphs to allow you to clearly see how execution goes.
Where’s the catch: the cons of Katalon
Poor language support. For now, you can write scripts only in Java and Groovy to be used by Katalon.Small (although growing) community. Even though its knowledge base is extensive, it can’t compete with Selenium in number of users. Which means you’ll have a hard time finding updated articles and reviews.
When does Katalon Studio succeed?
Katalon is actively used by startups, small and medium teams with limited tech capabilities. Even expert testers can find the tool massively useful with its long list of features and an advanced scripting mode. However, if you’re not using its paid dedicated support, Katalon is not suitable for large companies. Big projects require better and stronger support along with a more mature community, which Katalon can’t yet provide.Unified Functional Testing (UFT)
UFT, originally called QuickTest Professional, is a commercial tool by MicroFocus that was acquired by Hewlett Packard in 2006. In its Magic Quadrant for Software Test Automation 2017 report, Gartner named it one of the leaders on the market, followed by Tricentis Tosca. This collaborative and easily integrated tool allows you to execute testing using both keywords and manual scripting. It supports CI tools, provides web, mobile, and API testing, and has a smart object recognition function.Why it’s good: the pros of UFT
Automated tests from manual. UFT makes manual testing easier by allowing developers to record previously manual tests and export them as automated ones. With its Sprinter software, you can convert execution reports to automation test cases.Collaboration capabilities. UFT has a shared repository where members of you team can store artifacts, functions, and spreadsheets.
Where’s the catch: the cons of UFT
Only one language supported. UFT uses VBScript as its scripting language. However, having a reputation as one of the easiest languages to use and code with, it may not be such a big con, especially for pros.Price. The tool is one of the priciest on the list, considering it doesn’t allow for testing desktop apps like TestComplete does. Despite being less than reasonable, it can be a huge stepping stone for many developers. A year's seat-based license will cost you $2,500 and a concurrent one - $3,500. UFT doesn’t offer any group discounts though they may provide one in their custom enterprise package.
Only Windows support. Just like several other testing tools that we’ve featured, UFT can’t be used on a Mac without additional software.
When does UFT succeed?
Big teams can leverage its collaboration and web/mobile/API testing features when working on a larger project. To benefit from UFT and justify its price, you need an established testing framework or must be ready to set one up.Watir
Despite its open source and basic nature, Watir (Web Application Testing in Ruby) is used by such huge companies as Slack, Yahoo!, and HomeAway. Watir is limited to web apps, works only on Windows, and is targeted to executing simple and easily maintainable tests.Why it’s good: the pros of Watir
Choice of languages. Watir is one the few tools that allows writing scripts in Ruby programming language. Ruby is very popular for testing since it’s user-friendly, easy to learn, and fast to code with. That’s why it may be a big plus from Ruby developers. But if Ruby is not your language of choice, there’s also a way to use Watir. Script with Java or .NET and still leverage Watir’s simplicity with its sister-software: Watij and WatiN.Where’s the catch: the cons of Watir
May be too simple. Watir doesn’t try to be something more and it seems to be fine to those who swear by it. However, it can only be used in combination with other tools.Small community. There are not many complaints about the software online but there’s not enough written overall. Not only is it hard to find fellow developers, the documentation and contributed content is limited to what Watir lists on its website.
When does Watir succeed?
Ruby developers will find Watir very enjoyable to use (unless we’re talking about Watij or Watin). As you can see from the list of its users, companies of any size can successfully implement it in their testing efforts. If you want flexibility and are building your testing toolkit, Watir may be worth a try.Ranorex
Ranorex is one of the most expensive and most comprehensive tools we’re covering today. It supports web, desktop (Windows only), and mobile testing, a very long list of technologies and languages to choose from, automation both for inexperienced users and advanced testers, and customization with third-party frameworks.Why it’s good: the pros of Ranorex
Selenium WebDriver integration. You can make Selenium WebDriver in the Ranorex environment and leverage the capabilities of the biggest automated testing framework. Such close integration allows Ranorex to fix all pain points Selenium still has.Seamless integration with CI tools. To successfully use Ranorex in CI development, you can easily integrate it with Jira, Jenkins, Bamboo, or TeamCity, the most common continuous integration services.
Complex GUI testing. Visual interface testing is a complicated task usually delegated to manual testers. Ranorex automates it by using object recognition and following different user scenarios while freeing testing staff from repetitive tasks.
Price. It’s a fairly inexpensive tool for the amount of functionality it offers. You can grab it’s one user license for $2,800 and add additional machines and users for an extra $850.
Where’s the catch: the cons of Ranorex
No Mac support. Ranorex is yet another framework that doesn’t run on Mac OS and doesn’t allow for testing Mac apps.When does Ranorex succeed?
Brands like Siemens, Lufthansa, Cisco, Dell, and IBM trust Ranorex to automate their testing. With this tool, you can have an all-in-one environment that can be easily expanded with more popular technology. It might not be the first choice for a small development team, but companies with large budgets can safely invest in Ranorex.If you're interested in a more detailed description, read our article focused solely on Ranorex.
Final Takeaway
There’s a lot to consider when choosing a testing software since most of it is costly or requires additional learning. Use our guide to filter different options and find the ones your team would love to use. Make sure you request free trials of proprietary tools to allow your testers to ease into the workflow before the official testing begins.Let’s quickly recap the key points you and your team should consider when choosing an automation tool:
Platforms and languages. As you can see, not all options offer a wide range of scripting languages to choose from, so it may be a quick no from you to the tools that don’t offer your preferred technology. The market is especially small for Mac users, so you may want to consider getting a virtualization program.
Coding skills. You definitely want some flexibility when it comes to your testing staff. Even if you’re not yet planning to invite less skilled testers to your team, the time may come when you want your expert developers to work on more demanding projects. Select a solution that provides this freedom.
Continuous Integration. CI is becoming a norm in the development and testing world so it’s better to focus on tools that were built to work smoothly in an Agile environment. See which of your preferred scheduling tools they support and what other capabilities your team can make use of.