Everyone’s heard of the Y2K problem that occupied techies' minds at the turn of the millennium. Millions of dollars and hundreds of man-hours have been spent to fix the issue that engineers of the past had no chance of predicting.
A bit earlier, in 1998, a much more trivial software mistake was uncovered. The Mars Climate Orbiter, launched by NASA to study the Martian atmosphere, was suddenly destroyed as it approached way too close to the planet. All this happened because NASA used metric units to calculate the trajectory while the spacecraft manufacturer Lockheed Martin operated on imperial units.
This mistake is something that should have been uncovered in quality control. Different from quality assurance, quality control is something that’s done to ensure the defective product doesn’t make its way to the market. Let’s talk about what quality control entails in modern software development.
What is Quality Control?
According to ISO 9000, quality control is the process of fulfilling quality requirements.
Since humanity started manufacturing products en masse, we wondered: “How can we ensure that all customers are equally satisfied with the end product?” Defects and deviations from the original design have always happened, so different procedures were invented to check that the product is good enough to go to market.
Acceptance sampling, popular during World War II, would determine product quality by analyzing random samples. Control charts have been used since the 1920s to monitor the production process and determine the sources of variation.
In response to the variety of quality standards that government-contracted organizations had to comply with, in 1987, the International Organization for Standardization published now universally recognized quality requirements. ISO 9000 made it easier for clients and suppliers to specify quality expectations and the ISO 9000 certification allowed over a million worldwide vendors to confirm their product quality.
In software development, quality control exists alongside two other concepts: quality assurance (QA) and testing. Let’s examine them closer.
Quality Assurance vs Quality Control vs Testing
The lines between these three processes are somewhat blurry. The differences might not have much meaning for particular organizations or projects, who prefer to call all these processes QA or testing.
By quality assurance, we mean the set of actions aimed at preventing defects from happening. The main activities here include requirements analysis, planning, and documentation – basically, the tasks aimed at making it very clear what end product is expected from developers so that it’s accepted by users and stakeholders.
At the same time, quality control is what we do to verify that the product doesn’t have defects, meaning that it adheres to the requirements. This is done by testing the finished product or its parts for how acceptable they are.
Testing is comprised of activities that identify existing defects in the final product or parts of it. There are different levels of testing activities. There’s unit testing which examines the smallest part of the software and is typically done by engineers themselves. Then, there’s integration testing for checking how software units work together. And the entire system testing is done by professional testers and QA specialists who review whether the system complies with previously set quality standards.
Acceptance testing is the high-level test performed on software
But at the highest level, there’s acceptance testing, which is the main signifier that the product passes the requirements check. This is where most quality control activities reside. Unlike previous testing stages, acceptance testing can identify problems that may arise during user and business scenarios. Only acceptance testing can make sure that the product is ready to be shipped. To understand acceptance testing better and how quality is determined in software development, let’s cover the quality control process.
Quality Control Process
At its core, quality control is the process of verifying that the final result matches the expectations we had at the start of development. How do we do that?
Determine software requirements and acceptance criteria
The Agile philosophy of software development changed how we understand and approach quality. When the waterfall method prevailed, quality management in software was similar to that of manufacturing or construction. The focus was on fixing bugs as they happen and conducting testing only after the job had been done. With agile, testing has become part of each development iteration with a constant feedback loop allowing you to check the quality on a regular basis.
So, what is quality in the agile age? Today, quality is defined as an adherence to two notions:
functional requirements, or whether the software has features requested by a user or stakeholder, and
Watch this video for a visualized explanation of requirements
You can think of functional and nonfunctional requirements as task lists for developers to follow.
A user should be able to change the interface language is a functional requirement.
A system must make calculations in metric units only is a nonfunctional requirement.
As you may have noticed, these statements are devoid of any context or emotion. Why are these requirements important for product quality? What value do they bring? For developers to be able to work on values rather than features, user stories are written.
A user story is not a feature, it’s an end goal, expressed from the user’s perspective.
As a user, I want to operate the software in the language I understand.
As a user, I want the system to make calculations in the units I’m used to.
Phrased like this, stories shift the focus to the user and enable more creative solutions. But they’re not very handy for measuring quality. How do we check that the value was delivered? That’s why we have acceptance criteria.
This video will explain you all you need to know about acceptance criteria
Acceptance criteria describe specific scenarios or rules for how features must work. They’re often written in the Given/When/Then format.
Given that the user navigates to the account settings page.
When the user selects language settings.
Then the list of available languages appears.
Given that the user chooses their language.
When they agree to changes.
Then the interface language changes.
How requirements, user stories, and acceptance criteria relate to one another
If requirements are task lists, user stories are contextual signifiers, then acceptance criteria are specific checkmarks for devs to follow. They give unambiguous instructions on features expected by the user. And at the same time, they are useful to determine whether the systems passed or failed the quality test.
Consider acceptance testing types
The aim of acceptance testing (AT) is to verify that the product adheres to business needs in terms of goals, user acceptability, rules and regulations, operations, etc. Issues found during acceptance testing deem the product not ready for market and must be fixed immediately. Unlike other types of testing, AT requires real-time and production data. Here are the types of AT that are often performed for the final quality check.
User Acceptance Testing. Also called End-User Testing, User Acceptance Testing (UAT) is performed from the perspective of the end-user or by an end-user specifically recruited for the test. It checks only functional requirements and is of an uncontrolled nature, meaning that it has few restrictions on how the product should be used during the test. UAT can be performed not only at the final stages of development but during UX/system design or as soon as a product demo is ready.
Business Acceptance Testing. Business Acceptance Testing validates whether the product fulfills the business objectives and is ready for the needs and challenges of the real world, such as adherence to internal business processes, industry standards, economic feasibility, organization policy, etc. Here, both functional and nonfunctional requirements are checked. As the market is rapidly changing, the product, for which acceptance criteria have been written months or years ago, may be deemed unacceptable by the time it’s ready, so extra time could be needed to prepare it for the new reality.
Contract Acceptance Testing. This is the type of test performed to ensure that contract requirements were fulfilled. In the contract or the service level agreement, there are terms by which the payment must be made. Even if the product doesn’t pass user or business acceptance testing, it might pass contract acceptance testing. The contract usually has detailed terms on when and how the testing is conducted and what conditions are considered acceptable or not.
Regulations or Compliance Acceptance Testing. This testing checks whether the product complies with the rules and regulations of the countries where it will be released. Even if it passes every other acceptance test, failing regulations testing would mean that the use of the product just won’t be allowed to use.
Operational Acceptance Testing. Also called operational readiness testing, it checks non-functional requirements such as the product’s scalability, maintainability, compatibility, stability, and others.
Write test cases from the acceptance criteria
A test case is a basic set of actions performed to verify product quality. For acceptance testing, the Given/When/Then format used for writing acceptance criteria is a very effective format for a test case as well.
Here’s how an acceptance criterion looks.
Given that the user navigates to the account settings page.
When the user selects language settings.
Then the list of available languages appears.
Given that the user chooses their language.
When they agree to changes.
Then the interface language changes.
And here is how it can be easily converted into a standard test case format.
Transforming acceptance criteria into acceptance test cases
This connection between test cases and AC also simplifies iteration. If during the test, we see that the scenario doesn’t play out as expected, we can easily rewrite user stories and acceptance criteria.
Conduct automated acceptance tests
Although acceptance tests typically review business-facing requirements rather than technicalities, they can still be automated. Moreover, in the agile setting, they should be automated. Many AT automation tools understand the Given/When/Then format and can execute features with minimal tester input.
Here’s what writing an automated test using a popular tool Cucumber looks like.
Writing Acceptance Tests with Cucumber
Source: Semaphore
Other handy tools for automated acceptance test creation include JBehave, Behat, SpecFlow, and more.
Adopt Acceptance Test-Driven Development
There are three popular Agile testing methodologies:
Test-Driven Development (TDD),
Behavior-Driven Development (BDD), and
Acceptance Test-Driven Development (ATDD).
They are all based around the same concept of writing tests before any development, running them, failing, and then going back for adjustments. The difference is that they do that on different levels.
Difference between TDD, BDD, and ATDD
While in TDD, you start by writing test cases, in BDD, you begin by defining user stories, and in ATDD, you write acceptance tests first. The distinction between the latter ones is a bit vaguest, but ATDD has two unique goals:
ATDD is focused on defining the requirements correctly rather than checking if requirements match the system’s behavior. It ensures that the product’s quality from business and user perspectives are considered first.
ATDD involves all types of stakeholders from developers and testers to business and domain experts to share a full understanding of the project goals and have everyone involved in writing acceptance tests.
ATDD is a valuable technique for products in highly competitive markets, where customer experience is paramount.
Who Performs Quality Control?
The quality control process entails several different tasks, from being involved in user stories and acceptance criteria writing to preparing automated acceptance tests. So, who’s doing what in the product development team?
A product owner generally is a person responsible for the project outcome who serves as a customer stakeholder. They are the ones creating a product backlog where user stories live. To refine user stories and acceptance criteria, a product owner teams up with a business analyst, project manager, and sometimes a subject matter expert.
After that, the whole development team is invited to analyze acceptance criteria and brainstorm about how they can be implemented into features.
Then, programmers and QA engineers can start crafting acceptance test cases from acceptance criteria. They are also the ones who might initiate changes to acceptance criteria if they encounter the need for it during test case writing.
Acceptance tests themselves are performed by different stakeholders: customers, end-users, and specialized testers hired for the job.
Quality > All Else
Customer satisfaction is the synonym of quality in the agile paradigm. The first principle behind the Agile Manifesto reads, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
Delivering and maintaining quality as we understand it is an extremely difficult task, especially considering the competition, time, and financial restrictions many products must overcome as well. The best result lies in preparation and a deep-rooted understanding of what quality is to those who will be using and enjoying your product. Perhaps ATDD will help you achieve that level of quality.
We also recommend you watch our video series on software planning on YouTube for visual explanations of the most popular concepts.
