As you build a product, your codebase keeps growing and, unless properly managed, can become a virtual Rubik’s cube for future developers to solve. Back in the day, when waterfall methodology ruled, it could take months or even years to deliver a product's first shippable version.
Switching to Agile methods helped reduce programming cycles to weeks and introduced steady-interval delivery. Modern practice of continuous integration (CI) and continuous delivery (CD) rolls out program updates even faster, within days or hours. That’s the result of the frequent submission of code into a shared repository so that developers can easily track defects using automated tests, and then fix them as soon as possible.
Learn more about benefits of continuous integration and delivery, how to approach its adoption, and what challenges to expect along the way from our dedicated article.
Note, though, that CI/CD isn’t some magic wand that automatically fixes your workflow. To make this miracle happen you need a corresponding tool. Take an easy route and read this article for a clear picture of the best-of-breed options. We hope you'll find answers to practical questions like: What CI/CD tools can do? How hard is it to learn? What will it cost me? Will it work with my infrastructure? And, finally, which one is the right fit for me?
Components of a CI/CD tool and how it works
All CI/CD tools do the same work: They run mundane, repetitive tasks to safely ship iterative code updates to end users. Their functionality covers
- build automation, where build means turning source code into a deployable version,
- test automation, and
- deploy automation.
A CI/CDl solution consists of two distinct parts: a central CI server also known as a master and one or more build agents often called build slaves, build runners, or remote agents.
How CI/CD tools automate build jobs.
The CI or central server is responsible for assigning and orchestrating jobs and running them through . It also tracks the executions and displays their results. The server is integrated with a version control system (VCS) so it can detect changes in the source code and trigger a pipeline — a predefined sequence of tasks to be performed . Despite the CI part in the name, most of such platforms are not limited to integration and also support continuous delivery and deployment.
Build agents are remote individual processes that listen to the server and execute its tasks. They can run jobs on schedule or in response to certain events — say, once you commit a new portion of code to a repository on GitHub, GitLab, or other popular VCS. Tasks can be executed in parallel, by several agents. This capability is especially useful when you deal with a huge codebase or have to test your update on different operating systems.
That’s how CI/CD software generally works. Now, let’s see what options you have and how to choose the best one.
How to choose a continuous integration tool
Today’s market is flooded with continuous integration and delivery solutions, so selecting the right one can be difficult. To guide you through the many options on the way to making a choice, we suggest using the following criteria:
Most praised features
As we said earlier, CI/CD tools cover the entire process of carrying code updates from a version control system to end users. However, some options better handle integration jobs, some focus on testing and debugging, and others outdo its counterparts in deployment. Carefully read specifications and user reviews to understand the strong and weak sides of a particular solution
Hosting options
CI/CD tools can be hosted either on your server (on-premise) or on the side of a cloud provider. In the first case, the responsibility for deploying and maintaining them rests solely on your shoulders, or rather on your in-house DevOps team. At the same time, on-prem tools benefit from building flexibility and security.
Cloud solutions are available to end users via the Internet and fully managed by the vendor or cloud provider (which can be the same company). This option spares you the setup hardships, offers great scalability and can be adjusted on-demand.
Сompatibility and integrations
Compatibility of new software with technologies you already use in the development lifecycle will save you a lot of headache. Check whether the CI/CD solution of your choice supports and easily works with your current
- operating system (OS),
- project management tools (e.g., Jira),
- integrated development environment or IDE (e.g., Visual Studio),
- continuous testing software (e.g., Katalon Studio),
- version control system (Git, GitLab, Perforce, etc.), and
- cloud services.
It’s worth noting that some CI/CD tools are tied to a particular VCS, cloud provider, and/or virtual machines limiting your deployment options. The same goes for an operating system. To preserve flexibility, look for a platform-agnostic solution.
Support for Docker containers
Support for Docker can mean two different things: using Docker containers for deployment and for build tooling itself. While the former is something customary, the latter is not as conventional. Traditionally, CI/CD platforms were installed on a virtual machine (VM), a digital version of a physical computer. Today, many of them allow for running jobs in the Docker environment which brings several advantages.
- Reusability. Container-based pipelines allow you to reuse their blocks in other workflows.
- Versatility. You can rely on the same software stack across all platforms.
- Portability. As the de facto delivery standard, Docker doesn’t tie you down to a particular cloud provider, OS, or framework.
Therefore, check if your CI/CD tool offers Dockerization right from the start.
Extensions and plugins
Plugins are created to extend or enhance the functionality of the software and support integrations with other technologies and environments. The more add-ons your tool has, the broader its capabilities. In that respect, a free and open-source solution is usually in an advantageous position since numerous users constantly develop new extensions and upload them to the publicly available repositories. Not all of those plugins are essential or really useful; however, if wisely picked up they can significantly improve CI/CD automation process.
User interface and learning curve
Some tools simplify a build process more than others thanks to their clear-cut user interface and build visualizers enabling you to manage and configure CI/CD pipelines (sequences of jobs) graphically. A well-designed UI/UX also significantly flattens the learning curve, making the adoption process go faster.
Pricing and availability of free tier
Last but not least, pricing policy can determine the case. Free tier offers give you an opportunity to try and test the product before investing. Otherwise, you always run the risk of acquiring a pig in a poke.
Now that important criteria are defined, let’s take a closer look at the most popular CI/CD tools.
Comparing the most popular CI/CD tools.
Jenkins: the most-used CI/CD solution with great community support
Works best for: anyone but it especially benefits projects, where you need a lot of customizations. You may change almost everything here, still, this process may take a while.
Jenkins is an open-source project written in Java that runs on Windows, macOS, and Linux, and can be installed locally or in cloud environments. Designed in 2004 for continuous integration, it now covers the entire CI/CD pipeline. The solution is free, community-supported, and might be your first-choice tool.
Main selling points
No expenses required. Jenkins is a free CI/CD tool and that can save you money on the project.
Limitless integrations. Jenkins can integrate with almost any external program used for developing applications. It allows you to use Docker and Kubernetes out-of-the-box. G2 Crowd reviewers claim: “No better tool exists for integrating your repositories and code bases with your deployment infrastructure.”
A rich library of plugins is available with Jenkins: Git, Gradle, Subversion, Slack, Jira, Redmine, Selenium, Pipeline, you name it. Jenkins plugins cover five areas: platforms, UI, administration, source code management, and, most frequently, build management. Although other CI/CD tools provide similar features, they lack the comprehensive plugin integration that Jenkins has. Moreover, the Jenkins community encourages its users to extend the functionality by providing teaching resources.
Active community. The Jenkins community provides a guided tour to introduce the basics and advanced tutorials for more sophisticated use of the tool. They also hold an annual conference DevOps World | Jenkins World.
Main weaknesses
Documentation isn’t always sufficient. For instance, it lacks info on pipeline creation. This adds time-consuming tasks to the list, as engineers have to work through them.
Poor UI. Its interface seems a bit outdated as it doesn’t follow modern design principles. The absence of whitespaces makes the views look crowded and confusing. A lot of the progress features and icons are super pixelated and don't refresh automatically when jobs finish.
Jenkins dashboard
Plugin issues. Since most plugins are developed by open-source contributors, they are not consistently supported. Nor are they tested together, which may lead to conflicts.
TeamCity: built-in features for almost everything
Works best for: enterprise-grade companies and self-supporting teams ready to build their own plugins when needed.
TeamCity by JetBrains is a reliable and high-quality CI/CD tool often chosen by teams for the rich functionality out-of-the-box, plus Docker support. It’s cross-platform and runs on all the recent versions of Windows, Linux, macOS, and other Unix variations (Solaris, FreeBSD, IBM z/OS, and HP-UX).
TeamCity works right after installation and boasts a number of unique features such as detailed history reports, instant feedback on test failures, and reusable settings so you don’t have to duplicate your code.
Pricing model. TeamCity offers a 14-day free trial for its cloud version and free on-prem tier with full access to all product features but limited to 100 build configurations and three build agents. Adding one more build agent and 10 build configurations costs $ 299. TeamCity also gives 50 percent off for startups and free licenses for open source projects. The license for cloud teams starts from $45 per month and depends on the number of active committers.
Main selling points
.NET support. TeamCity integrates with .NET technologies better than any other CI tool out there. There are many important .NET tools included in TeamCity, such as code coverage analysis, several .NET testing frameworks, and static code analysis.
Extensive support for different environments. TeamCity comes with native support for such popular tools as Jira, Visual Studio, Bugzilla (bug tracking), Maven (build automation), and NuGet (package managing). It works with a wide range of version control systems including GitHub, GitLab, and VCSs.
The cloud version of TeamCity originally runs on AWS but it has extensions for smooth deployment on Microsoft Azure and Google Cloud. Over 400 plugins are available now to set other integrations including Kubernetes.
Platforms and environments supported by TeamCity
Ease of setup. TeamCity is easy to set up and ready to work right after installation.
Built-in features. TeamCity provides a broad set of out-of-the-box functions such as
- build chains and dependencies,
- source control,
- status monitoring across multiple projects, and
- detailed history reports and statistics.
TeamCity shows the progress of the build at every step and the number of tests remaining to pass. It also lets you deploy the code to any environment.
Competent docs. The overall functionality can be easily understood by going through the provided user guide, which is thorough and extensive.
Main weaknesses
Steep learning curve. Although TeamCity is well-known for its visual-aesthetic UI, it still can be complex and overwhelming for newcomers. It may take developers some serious study before they are ready to use the tool in production.
Manual upgrading process. Moving from one major version to another is a long process that has to be done manually on your server.
Bamboo Data Center: out-of-the-box integration with Atlassian products
Works best for: enterprise-grade companies, already using Atlassian products.
Bamboo is a CI/CD tool by Atlassian with focus on the delivery aspect. It runs on Windows, Linux, and macOS. In February 2021, the company stopped selling licenses for their server products, with support for them ending in February 2024. By then, all their customers are expected to migrate to Atlassian Cloud or on-premise Data Center. Bamboo is now available only in a Data Center version. All existing customers must install new licenses before the end of support.
Pricing models. Bamboo is free to try for 30 days. After that, your options are limited to the enterprise-grade tier. The annual subscription covers unlimited jobs and software maintenance. The price will depend on the number of remote agents or build slaves. One remote agent will cost you $1,200 per year.
Main selling points
Native support for Atlassian products. Bamboo natively connects with other popular Atlassian products such as Bitbucket, Jira, and Confluence. In fact, the same integration for Jenkins takes a huge plugin scheme.
Improvements to the previous server edition. Bamboo Data Center enjoys significant enhancements if compared to the old server version. Among most valuable additions are
- clustering to ensure continuous uptime and disaster recovery. If one node in your cluster breaks down, you have others to take up the job almost immediately;
- improved build and delivery resilience meaning that the agent will finish its task even if the server is down. Once the work is done, the agent tries to reconnect to the server;
- support for a single sign-on that expands on all Data Center products, and
- the ability to run Bamboo on a Kubernetes cluster, which simplifies administrative tasks and brings agility across your teams.
All above-mentioned perks have been long available in TeamCity and Jenkins but the same couldn't be said for Bamboo. With moving to the Data Center version, the tool will become more viable.
Ease of deployment. Releases are streamlined by out-of-the-box connections with Docker and AWS CodeDeploy, a service that automates deployments across development, test, and production environments.
Documentation and backing from Atlassian. Bamboo documentation is rich and detailed. Atlassian provides skilled support. However, community size is far from Jenkins’ user reach.
Main weaknesses
Poor plugin support. In contrast to Jenkins and TeamCity, Bamboo doesn’t support that many plugins. There are only 185 apps currently listed on the Atlassian repository.
No cloud version. Unlike other similar solutions, Bamboo doesn’t have a fully managed cloud version. You have no other choice than to run and maintain it yourself on your own servers or using resources of cloud providers (primarily, AWS.)
High price, with no license for small companies. The price of a license starts from $1,200 for just one agent. So, Bamboo Data Center is hardly affordable for small companies.
Travis CI: a multi-language, easy-to-setup CI solution
Works best for: quick, small projects.
Travis, one of the oldest CI solutions, has won the trust of many customers and powers well-known projects like Ruby on Rails, Ember.js, OpenSSL, and Puppet. It has a large and helpful community that welcomes newbies and provides a great number of tutorials. The tool can be set up in one of three environments: Ubuntu, macOS, or Windows. You can run your builds in Docker containers but this option is currently unavailable for macOS.
Travis eliminates the need for a dedicated server, as it is hosted in the cloud. However, it also has an on-premises version for enterprise customers.
Pricing models. The core license is based on the number of credits which are units for purchasing build minutes. It starts from $30/month for 50,000 credits. You can test how many credits you need for your jobs by using a 30-day free trial that comes with 10,000 credits. There is also an enterprise license based on custom agreements.
Main selling points
Easy setup. Travis CI requires no installation. All you need to get started is to sign up with one of the preconnected VCSs — Assembla, Bitbucket, GitHub, or GitLab.
Support for multiple languages. Travis CI is a real polyglot in the world of continuous integration and delivery. It supports over 30 programming languages, in the alphabetical order from Android to Visual Basic if arranged.
A wide range of pre-integrated deployment options. Travis CI automates continuous deployment to 40 platforms, You can find the full list of possible options here. If you want to deploy on your own server, add a custom step. Code shipping on the unsupported platforms is available via a deployment script.
Main weaknesses
Limited plugins and third-party integrations. Travis CI lags behind Jenkins and other tools in the number of plugins and third-party integrations. This makes the tool less flexible and less customizable.
Issues when handling large projects. Working great for simple and small deployments, Travis CI loses its competitive advantages with codebase growth. It becomes hard to configure as complexity of the project increases and specific requirements arise.
CircleCI: a mature platform with a plethora of pre-built integrations
Works best for: tech-savvy teams looking to speed up CI/CD processes.
CircleCI is one of the world’s most used CI/CD platforms, processing over a million builds daily. Among its clients are Spotify, Coinbase, Samsung, and Docker. It comes as a software-as-a-service offering but also can be installed on your hardware or private cloud. The tool natively runs on Windows, macOS, Linux, and Docker environments as well as on ARM processors that are installed in many mobile devices (smartphones, tablets, wearables, etc.)
Pricing models. CircleCI sticks to a ‘pay-as-you-go’ pricing model. You can start for free if you need no more than 6,000 build minutes per month. Once you exceed this limit, the price starts from $15 per month and will depend on additional time you actually spend and the number of users with five already included in the initial plan. The enterprise-grade tier will cost you at least $2,000 per month.
Main selling points
Convenient dashboard for job monitoring. CircleCI is recognized for its feature-rich analytics dashboard that allows you to monitor and analyze your jobs in real-time.
CircleCI dashboard
Numerous out-of-the-box integrations. Besides native support for GitHub and Bitbucket version control systems, CircleCI boasts pre-built integrations with popular
- services from major cloud providers (AWS, Google, and Microsoft Azure), languages
- collaboration tools (Jira, Slack, Twilio),
- build tools (Maven, UBX),
- testing tools (Oxygen, Unmock, Blackfire.io),
- security tools (Wallarm, Snyk, SecretHub, Cypress),
- deployment tools (DeployHub, Helm, OpenShift)
- monitoring tools (Datadog, Honeycomb)
- code analysis tools (Codecov, CodeScene),
- languages and frameworks (Android, Flutter, Hugo, Node.js, Rust), and
- many more.
For now, the total number of connections amounts to 150. You can look through the full list here.
Reusable configuration packages. CircleCI has a rich library of reusable and shareable configuration packages called Orbs. These packages are created either by the CircleCI team, their partners, or community members. Anyway, existing Orbs can significantly speed up and simplify setting up your own pipelines. You can also write your own Orbs and reuse them across your project.
Ability to quickly find answers to any question. An extremely popular tool, CircleCI takes advantage of the detailed tutorials created around it. You can find answers to practically any question in the CircleCI documentation (or just by asking other developers.) If your problem remains unsolved, you can submit a request on the support page. They typically respond within 12 hours.
Main weaknesses
Technical complexity. To start working with CircleCI, you need to know YAML (Yet Another Markup Language). While it’s a common technology for building CI/CD pipelines, other tools usually offer a visual design feature as an alternative method to manage workflows with a no-code experience.
Unpredictable prices. As your project grows, so do expenses. With the per-minute and per-seat pricing model, it’s often hard to predict how much you will spend by the end of the month.
CodeShip: a cloud-based CI tool with fast builds
Best works for: small and enterprise-grade projects alike.
CodeShip by CloudBees is an AWS-based software-as-a-service solution for running CI/CD. By means of parallel pipelines, concurrent builds, and caching, CodeShip allows for deploying with ease multiple times per day. It enables you to quickly scale your cloud resources (CPU and memory) as you grow.
CodeShip dashboard
Pricing models. CodeShip is available in two versions: Basic and Pro. The Basic version offers pre-configured CI dependencies and a simple web interface but without Docker support. It comes in several paid packages from $49 a month to $399: The more parallelization power the package has, the higher the price tag. The Pro version supports Docker and is more flexible. You can pick your instance type and parallelization up to 20x.
There’s also a free plan limited to 100 builds/month, one concurrent build, and one parallel test pipeline. Educational projects and non-profits get 50 percent off. Open source projects are always free.
Main selling points
Centralized team management. CodeShip allows for setting up teams and permissions for your organizations and team members. Currently, they’re offering the following roles: owners, managers, PMs, and contributors.
CodeShip role permissions
Parallel tests. CodeShip has a special Parallel CI feature for running tests in parallel.
Simplified workflow. Intuitive UI for effortless configuration and file management get the workflow going quickly. Almost all G2 Crowd reviews point out that CodeShip is very fast and easy to get started with GitHub and Bitbucket. They say that in CodeShip all things are much easier and the deploying process much simpler than in Jenkins.
Code shipping with one button. CodeShip makes it easy to continually push out code to all the environments in one click.
Reliability. “CodeShip allows us to have peace of mind with every push, to ensure that all tests and any other necessary preprocessing are completed every time,” says a review at G2 Crowd.
Main weaknesses
Lack of documentation. Based on reviews, there’s a lack of documentation, especially in comparison with Jenkins, and it should be extended.
Relatively small number of native integrations. CodeShip's list of pre-built integrations is quite short in comparison with similar tools . Yet, it includes almost everything you may need for your CI/CD process — say, VictorOps and Opsgenie for alerts and notifications, SourceClear for automated testing and reporting, Snyk for security checking, Raygun for tracking error logs and deployment events, Codacy and Codecov for analyzing code quality, and many more.
Final recommendations on choosing a CI/CD tool
How to select the CI/CD solution that will not only meet your current integration requirements but also evolve along with your product’s future roadmap? Going through the following checklist will make your choice easier:
Listen to your team’s needs. It’s important that your team can quickly learn how to use the chosen tool and start developing the product. Depending on the expertise level of your engineers and the programming software they already work with, the range of CI tools can be narrowed down.
Keep in mind the functionality you need. Jenkins is a great solution for continuous integration, but it’s not that pointful if you already have a CI system and look for a CD tool. Smaller tools like Spinnaker are great for testing and delivery, but not meant for integration. These examples show that each tool is good at some particular functions. So first you have to figure out your primary business goal and what functionality can satisfy it.
Know your company's data storage rules. Although it’s very convenient to leverage cloud-hosted services, sometimes it is not reasonable to hand infrastructure management to a third party. Legal limitations could turn out to be a deciding criterion if your company deals with data that is subject to strict regulations — like protected health information (PHI). In this case, it’s better to store data locally, on an on-premises server or on a private cloud.
Make sure the solution fits your budget. Consider your project’s size and go for an option that satisfies its needs at a reasonable price. For example, startups can make better use of the cloud-based CI/CD solution, as it includes the basic essential features for a small number of users with minimal or no fees.
Open-source CI tools are mostly community-driven with plugins and support available via online tutorials, blogs, chats, and forums. However, if there’s a need for support for pipeline maintenance and no budget constraints, pick a proprietary option. Alternatively, you can stick with an open-source tool, as long as there are organizations offering commercial support for it.
Test the workflow with different CI tools. It’s very rare that a single CI tool will suffice for all scenarios. Even the prominent CI tools covered in our article can hardly meet 80 percent of the automation requirements since projects range from monolithic, monster software systems to microservices-based architectures.
A good strategy is to use multiple CI tools for different needs instead of struggling to fit all in one tool. This approach will also contribute to business continuity, securing projects if a CI tool is discontinued or its support turns out to be insufficient.