Every application you build today relies on APIs. That means it’s crucial to thoroughly verify APIs before rolling out the product to the client or end-users. Some QAs, however, still ignore this layer of the test pyramid and proceed right to UI testing – at their peril. In this article, we explain why you should do API testing and how to approach it.
We have already written in-depth on what an API is. We’ve also elaborated on the basics of software quality management. This time we connect the dots under the topic of API testing.
What is API testing?
API testing is a set of quality assurance actions that include sending calls to the API, getting output, and validating the system's response against the defined input parameters, in particular, the accuracy of data and data's format, HTTP status codes, and error codes.
Usually, API testing is performed on APIs produced by the in-house development team. We don’t test third-party APIs, but we can test the way our software accepts their requests.
The approach to API testing largely depends on the API type. There are web APIs aka web services, database APIs that connect applications with DB management systems, operating systems APIs, and remote APIs for accessing resources located outside the device requesting them.
In this article, we’ll focus on in-house web APIs and their quality assurance.
API in terms of the three-tier architecture
Client, server, and database are the three independent tiers of software architecture. Usually, a client is a web browser or a mobile application that presents the requested info, while business logic is realized on the server side. Client and server communicate through requests based on different transfer protocols. The system and logic of protocol-dependent requests are the API itself.
The role of API in the three-tier architecture
In the most popular software architectural style for data transfers - REST - clients make HTTP/HTTPS requests to the server. The server retrieves the requested data from the database, transforms it according to the business logic, and returns it to the client as a response in JSON, XML, or other formats. Finally, the UI displays it to the user.
Another type of web API is SOAP API - a legacy web communication protocol that is still in use. SOAP isn’t limited to HTTP/HTTPS protocol but supports many others including TCP, SMTP, and FTP, but it works with XML format only. You can find out more about SOAP and how it differs from REST in our dedicated article.
And now, let’s see how it works from the QA side.
Why API testing is important
A basic concept in automated testing is a test pyramid. It demonstrates how to tackle automated testing for a project: what parts to test first as they form the base of the pyramid, and what to leave for the final stages - the pyramid’s tip.
Following the pattern of the testing pyramid, the first is the unit test layer - code testing often performed by developers while writing it. Then comes the server layer where API testing belongs. Later, when the front end is finished, UI testing takes place.
Three layers of the test automation pyramid
As we already mentioned, API testing is often overlooked. But why is it so important?
UI tests can’t cover every aspect of back-end testing and turn out to be insufficient for verifying functional paths and API services. This could leave you with bugs rooted in the server or even at the unit level. But now, the cost of mistakes is way higher as it may require rewriting a considerable amount of code and derail the release.
The solution here is to start testing at the early stages. If the request doesn’t return the needed value at the server layer, it won’t be displayed at the UI layer either. Not relying on UI to be ready, API tests can be performed early in the development cycle. This way, they’ll kill at least half of the bugs before they get serious.
In the Agile processes, where instant feedback is needed, unit and API testing are preferred over GUI testing as they are quicker to create and easier to maintain. While the latter requires considerable rework to keep pace with frequent change.
But don’t think we dismiss UI testing. Certain defects can only be caught by GUI tests and it’s the only test type that truly checks the user experience. We just encourage you to pay attention to API testing as well and apply it to smooth the overall development workflow.
Types of API testing: What they focus on
QA engineers approach API testing from different angles. They test whether it returns correct outputs in the expected format, whether it delivers responses within an acceptable timeframe, and how well it integrates with presentation layer software. Testers check an API’s reactions to edge cases (e.g. failures, unexpected or extreme inputs) and potential security attacks. So let’s address each of these aspects.
Functional API tests
Functional testing is the assessment of specific functions within the codebase. It makes sure the API actually works within the expected parameters, namely:
- returns the desired output for a given input; and
- handles errors when the results are outside of the expected parameters.
One of the functional testing types is Positive / Negative testing. Negative testing checks how an API responds to every possible kind of wrong input, while positive testing verifies the correct functioning of the API when the input conforms to the norm. If positive test cases fail, it’s a bad sign, as it means the application can’t perform even under ideal conditions.
API performance tests
To ensure the API can handle the expected or higher load, QA engineers validate its functionality and performance by artificially creating or simulating API calls. We'll outline the types of API performance testing.
Load testing. The point of load testing is to measure where the limit of system performance under high load lies. That’s why we measure response times, throughput, server conditions, etc., while increasing the number of calls.
Soak testing. Load tests that run over a long period of time can reveal system instabilities like API memory leaks. So when you have a weekend ahead, leave automated soak tests running. On Monday, it will show you whether any unwanted behavior has emerged.
Stress testing. The idea is to gradually increase the count of virtual users to find the point at which the API starts throwing errors, slows down, or stops responding.
Spike testing. Unlike stress testing, here an API undergoes a sudden and extreme increase or decrease in the number of users over a short period of time. Spike testing checks whether the API is able to stabilize and return to normal functioning after that.
Scalability testing. You want to be sure that your system performance scales according to the changing load. To do so, increase the number of incoming requests and monitor whether it causes a proportional increase in response time.
API security tests
Security, penetration, and fuzz testing are the components of the security auditing process aimed at testing an API for vulnerabilities from external threats.
Security testing. It validates whether security requirements are met. This includes authentication, permissions, and access controls, namely:
- what type of authentication is required to use the API;
- how well sensitive data is encrypted; and
- what authorization checks are set for resource access, etc.
Penetration testing. Taking security testing a step further, in penetration testing, certain API functions, resources, processes, or the entire API is under attack from the outside. This determines whether the threat vector can be reached.
Fuzz testing. The last step in the security audit tests the API at its absolute limits. Forcibly inputting massive amounts of random data, it tests whether the API will stand it or end up with negative behavior like a forced crash or overflow.
API integration and reliability tests
Being at the center of integrations between internal or third-party services, APIs need to pass the following tests.
Integration testing. Focused on communication between APIs, it ensures they are well-connected and don’t cause bugs in other APIs.
Integration API testing, Source: SoapUI
Reliability testing. When synced with various devices, an API should be tested for any possible disconnections.
How to go about testing the API
API testing is performed by submitting requests to the software through the API and then checking if it returns the expected data. When automated, API testing can be easily performed on a regular basis. The main advantage of automation is speed - less lag time between development and QA and less time spent on debugging in production.
QA engineers perform automated API testing using a testing tool that drives the API. We will discuss the most popular solutions in the next section.
API testing step by step, Source: Capital Terefe
Step 1: Create API testing requirements. To be able to plan API tests, first, you need to determine testing boundaries and requirements which include:
- the purpose of the API and its target consumer,
- the workflow of the application,
- the features and functions of the API,
- other APIs this API interacts with,
- aspects of testing,
- the problems we’re testing for,
- priorities in testing,
- the desired output, and
- the definition of Pass and Fail.
Step 2: Establish the API test environment. When you determined the functional scope of your API, the next step is setting up an API test environment which is usually the task of DevOps engineers. It requires the configuration of the servers, databases, and every resource the API interacts with, depending on the software requirements.
Step 3: Make a trial API call. Before diving into thorough testing, make an API call just to check that nothing is broken and the API is operational.
Step 4: Define the input parameters. Plan all possible input combinations. You’re going to use them in your test cases, and later - for the authentication of the results to determine whether the API performs as expected.
Step 5: Create API test cases. After all the preparations are done, you can write and execute test cases, and, in the end, compare actual results with the expected ones. A good practice is to group them by test category. Examples of API test cases are:
- testing value in response to the input condition: You define the input and authenticate the output, which can be any type of data or status (e.g. Pass or Fail);
- checking the behavior of the API in the system when there is no return value in response;
- tracking events or interruptions triggered by the output;
- authenticating the effect on the system after data structure updates; and
- validating resources modified by the API call.
API testing tools: top choices
API testing isn’t very different from any other testing. But, it requires specific tools, dedicated to automating it. Among other things, those instruments vary in the programming language they’re based on.
Using an API testing tool, QAs usually either take advantage of out-of-the-box solutions or develop a customized framework from its components.
A brief comparison of popular API testing tools
Postman API testing: REST API checks using JavaScript
Postman is currently one of the most popular tools used in API testing. Started as a browser extension for API validation, today with integrated test automation features, Postman is much more than just an HTTP client.
You can test APIs manually using Postman even with close to zero JavaScript skills. Igor Pavlenko, a Senior QA Engineer at AltexSoft, who is an avid Postman user, claims: “The tool requires only limited knowledge of JavaScript. So, basically, any Manual QA can create automated API tests just by using Postman predefined snippets”.
For automated QAs with extensive knowledge of JavaScript, Postman has a JS testing library.
Verifying the status code to be 201 via Postman
Postman advantages. The tool is praised for ease of use and speed, tons of features, and many integrations. It has a friendly UI for constructing requests and reading responses, allowing automated tests to be created quickly. Postman runs on local machines so that you can stay in control of your data. Using its command-line tool Newman, you can integrate these tests in continuous integration (CI) environments.
However, Igor also warns that it’s hard to maintain extensive API collections and its built-in versioning system is weak, so, “It is quite painful to work on the API tests together with the whole team.” Igor’s recommendation for large projects is to create powerful API tests with additional libraries like Requests (Python) or REST Assured (Java), which we’ll describe in a bit.
Postman pricing. Postman has four pricing plans:
- Free version for small projects with up to three users
- Basic version (from $12 per user/month) for fast API designing, development, and collaborative team testing;
- Professional version (from $29 monthly) for the centralized management of the API workflow; and
- Enterprise version ($99 monthly) for API-first development.
Licenses differ not only in the size of the feature list but also in the number of available monthly API calls to Postman API — from 1,000 in the free subscription to 1 million in the Enterprise version.
Requests: a human-friendly HTTP library in Python
Requests, an open-source Python HTTP library, was released under the Apache2 License with the aim of simplifying HTTP requests.
Testing the response time to be less than the timeout via Requests
Requests advantages. The Requests library saves time and effort by fully automating keep-alive and HTTP connection pooling. No need to manually add query strings to URLs and form-encode POST data. Igor Pavlenko considers Requests a powerful library with easy-to-understand documentation, simple syntax, and rich functionality.
“It supports such features as OAuth 2 and Open ID Connect Authentication, various Certificates and much more. So, the choice is simple: If you have Python Requests is the best decision for you.”
REST Assured: a Java library supporting XML and JSON requests
REST Assured is a Java library for creating a REST API testing tool script. So, to get started with it, you’ll have to set up a new Java project first, and then include it as a library for your project. Its documentation is stored inside the GitHub repository. REST Assured is the best fit for functional testing of REST API services and it requires strong coding skills to create tests.
REST Assured based test to check a GET API response, Source: Saurabh Agarwal
REST Assured advantages. API testing is pretty hard in Java, and Rest Assured was designed to simplify things. With it, you don’t have to create boilerplate code to test and validate complex REST responses. The library is open-source and has an active development community - another big pro.
JMeter: a Java-based tool measuring SOAP/REST API performance and more
Apache JMeter is a time-tested tool initially used for load testing. These days, Apache JMeter also supports functional, regression, and stress tests on different protocols. It’s a free open-source tool with custom scripting functionalities that require advanced programming skills.
Using JMeter for testing, Source: Saurabh Agarwal
JMeter advantages. JMeter's pluggable timers allow for simulating various types of loads. Its plugins add extensibility and customization, while built-in functions enable dynamic inputs to a test. Thanks to its efficient UI design and lightweight component support, JMeter executes tests fast and provides accurate timing. It caches test results and data providing their offline replay and analysis.
Oleksandr Fomenko, a Senior Test Automation Engineer at EPAM Systems, chose JMeter to measure the API performance for its convenient way of multi-threading emulation. “You can create test cases using JMeter’s record and playback technology, extract and reuse data from response, verify it using JSON Assertion, debug it, and build a comprehensive report. With these functions in stock, JMeter can be easily used as a stand-alone framework for testing REST APIs.”
RestSharp: simple RESTful API testing in C#
Available as a NuGet package, RestSharp is a popular and convenient open-source HTTP client library for . NET. It’s easy to install and add to your C# project. RestSharp parses both XML and JSON formats.
API test using RestSharp: Check whether the HTTP GET call returns status code 200, Source: Bas
RestSharp advantages. RestSharp’s functionality allows for straightforward test creation, serialization, and deserialization. It can handle synchronous and asynchronous requests with a wide list of ready-made authenticators. Uploading files and forms in multiple parts, RestSharp cuts down on upload times.
EPAM’s Oleksandr Fomenko highlights its powerful features and considers RestSharp the best solution for processing HTTP requests. He also accents its clean implementation: “RestSharp allows you to easily implement any type of HTTP client.”
SoapUI and ReadyAPI by SmartBear: end-to-end testing tools for REST, SOAP, GraphQL, and other web services
SoapUI is an open-source Java-based API testing tool by SmartBear for functional, security, load, and interoperability testing of SOAP, REST, GraphQL, and JMS (Java Message Service) web services.
Its commercial version - ReadyAPI (former SoapUI Pro) - covers more types of functionality tests (load, stress, and spike). It also supports the validation of event-driven microservices powered by Apache Kafka. Ready API comes pre-integrated with Git, Docker, Jenkins, TeamCity, and other popular DevOps tools. (We have already described another SmartBear product used for UI testing - TestComplete - in our article on automated testing tools)
SoapUI and ReadyAPI advantages. Both open-source and pro tools feature an easy-to-use drag-and-drop interface enabling scriptless creation and running of even the most complex test scenarios. They cover the full API testing spectrum. Yet, ReadyAPI has more extensive functionality including reporting and analytics. Also, as we mentioned before, it comes with out-of-the-box plugins for popular CI/CD tools.
ReadyAPI pricing plans. ReadyAPI offers three types of annual subscriptions called modules, each covering a different set of features:
- API Test Module starts at €755 (around $796) and includes end-to-end functional and security testing. It also gives access to a limited number of functional testing and virtualization instruments;
- API Performance Module starts at €6,010 (around $6,339). It you to validate the speed, scalability, and performance of APIs, servers, and network resources. It doesn’t include automated security testing, endpoint scanning, and some other instruments available in the API Test Module;
- API Virtualization Module starts at €1,131 (around $1,192). It allows you to virtualize critical APIs to reduce dependencies and delays throughout the delivery cycle. At the same time, the module lacks many testing features.
Each option has a 14-day free trial. You can also bundle all three modules and get the entire range of features at a lower price.
Katalon Studio: an automation tool with an inbuilt API testing mode
Designed primarily for UI testing, Katalon is also an integrated environment meant to easily generate and execute API tests.
Katalon advantages. Its edge over other automation tools like Selenium or Ranorex is that it doesn’t need additional integrations to test APIs. Katalon has an inbuilt API testing mode with the following capabilities:
- code auto-completion, inspection, and snippeting;
- references and debugger;
- inbuilt response viewer with auto-formatting;
- search for access to artifacts; and
- tests importing from API testing and editing tools (e.g. Swagger, Postman, WSDL).
With Katalon, you can test all types of REST, SOAP/1.1 and SOAP/1.2 requests and multiple data sources.
Katalon dashboard for testing API
Katalon pricing. There are three pricing plans:
- Free - for learning and experimentation. It covers 2,000 monthly test results;
- Premium - for production-grade testing workloads (from $25 monthly); and
- Enterprise - for large companies with mission-critical testing workloads, producing over 30,000 test results a month. Contact Katalon Studio for more pricing information.
If Katalon has piqued your curiosity, we have an article with an objective review of this software.
API testing framework vs ready-made API testing tool
Choosing among the tools we described, keep in mind that some of them are better for writing an automation framework (RestSharp, REST Assured, and Requests), while other tools (Postman, SoapUI, Katalon, and JMeter) can be utilized as they are.
So, which path to take?
Writing your own framework gives you more freedom as you’re not limited to the capabilities of a tool and its existing plugins. You can customize it your way: Add any library suitable for the chosen coding platform, introduce complicated logic in your tests, and build a reporting standard convenient to you. You also get more flexibility in managing your test data. But you enjoy these perks only if you possess strong coding skills.
Otherwise, you’d better look towards a ready-made tool. Its user-friendly interface and low-threshold coding requirements will be a relief for you. The pitfall here is that they usually address general API testing issues. But the good thing is that their community can easily help you solve your more-or-less common problems.
To sum up, a ready-made tool like Postman is good for starters. Developing your own automation framework is a more advanced level, but it’s also rewarding in terms of functionality you can add to your testing solution.