DevOps practice asks for two things. The first is merging development folks with operations folks (hence the name) into a cross-functional team with shared responsibilities. The second is automating as much as you can. We’ve discussed the former a lot. If you’re new to DevOps, check our video explainer below or get yourself something to eat and drink and binge-watch the entire YouTube series on that.
DevOps explained in less than 8 minutes
In the meantime, let’s talk about automation. The 2021 State of DevOps Report shows that successful DevOps practitioners on average spend less than an hour to advance a piece of code from the first commit to production. Laggards will need up to 6 months for that. The magic happens inside a continuous integration and continuous delivery (CI/CD) pipeline, a chain of systems that grabs the code that a developer has just committed, then builds it, runs it through test gates, and -- if everything works -- deploy to production.One of the major links in this chain is automating the build routine. So today we’ll discuss what build automation is, how it works, and the tools that can help you handle the process.
What is build automation?
The build is a process of preparing source code for production so that users can happily interact with it. It may include compilation, file compression, updating database schema, creating an installer or executable file, etc. That said,Build automation is an approach to handling builds within a CI/CD pipeline that has several steps. A developer commits source code to a repository, a CI server detects the change, runs the build process outside of the developer’s IDE (on a dedicated cloud or in-house machine), checks it with unit tests, and either returns for fixes or sends it further down the pipeline.
Automation helps standardize builds, make them faster, and avoid awkward mistakes typical for humans doing things manually.
How does that work?
Build automation process
As mentioned before, build automation is a part of a larger CI/CD pipeline in DevOps.
Build automation pipeline in a nutshell
So, let’s see how the committed code gets built with CI/CD.0. A developer commits code to a repository. The commit term comes from the realm of version control systems (VCS), services that help manage updates to the software. These may be GitHub, GitLab, Bitbucket, etc. A code commit operation means that an engineer sends new code to a repository residing inside one of those VCSs.
1. CI server detects changes inside the repository. A CI server is a system that orchestrates a CI pipeline, allowing teams to commit code multiple times per day, and build automatically outside of their workstations on different target servers. So, a CI server regularly polls a repository to check for new commits and save changes in a separate database.
CI server software allows engineers to tweak build configuration, add build agents that execute build jobs on dedicated machines, and integrate with build tools, which are specific for each programming language. What do these terms mean? Let’s keep exploring the process.
2. Build configuration recognizes changes and triggers a build agent. A build configuration is a set of rules determening how the build is supposed to happen. For instance, it sets the timing when the build begins, e.g.:
- Once a new code commit arrives
- On an established schedule, say, the team knows that by 5 pm every day they must commit their code, or
- After some external process is finished, like a security scan.
Build configuration sees changes in a CI server database and, based on settings, initiates different build agents.
3. Build agent activates build tools. A build agent is a program deployed on a build server (normally, outside of the CI server) that takes orders from the CI server and starts a build. It’s important that a single commit can trigger several parallel agents on different target machines, say, for different operating systems. Depending on a specific CI server, concurrent build agents’ work may be slightly different.
The agent doesn’t compile code, compress files, generate installers, or do any kind of low-level dirty work. It just listens to what a server with its configuration has to say and orders build tools to do their job in the way a developer configured. There may be a sequence of operations on the code that amounts to a build. The agent would be triggering those operations in a set order.
4. Build tools compile, compress, run tests, etc. Finally, build tools do their low-level work. A build tool is a script, framework, or any kind of software created specifically for a given programming language to compile code, run tests on it, or do other operations needed for a build. You’d expect your CI server to support integrations with such tools.
For example, Maven is popular for Java compilation and unit testing, you’d use MSBuild with C# and C++, to run tasks on JavaScript code you have Grunt, you get the idea.
At this stage, you’ll likely do unit tests, the lowest level quality assurance checks that verify that this particular unit or component works as intended. Some build tools already have unit testing support. Sometimes, an agent can initiate several build tools in a sequence to compile code with one tool and test it with another.
5. Build artifacts are sent back to the server. If the code passes all build stages including testing, you’d normally send it back to the CI server, at which point it can go to production, further testing, or other storage, depending on how your CI/CD pipeline works.
Build code comes in build artifacts, files ready to be deployed and executed (for example, .exe for Windows).
Full and incremental types of builds
There are two types of builds -- full and incremental.Incremental builds. Normally, the process takes less than 10 minutes as you don’t build the entire code base, but rather those parts of source code that have experienced changes. Build tools take care of finding the changed files.
Full or clean builds. Or you can take all code files and rebuild them from scratch. These builds are less common because they take a lot more time. In fact, incremental builds are so default that people google how to perform full and clean builds. But this still may be an option if you have stability issues and other specific cases, like removing obsolete files.
When it comes to automation, you expect to run incremental builds.
Popular CI servers
Now that you’ve got an idea how build automation works, let’s give the most popular tools a look. Keep in mind that the market of CI servers is quite mature and the differences that will tip the scales in favor of any tool will be subtle. All of them will get the job done. To get a more detailed overview, check our dedicated article on CI/CD tools.
Most popular CI servers
Jenkins: Open Source, Customizable, and Very Popular
If you don’t know where to start, Jenkins is your CI server of choice. It’s free, open-source, remaining the major and the most mature tool with the largest community out there. Jenkins UI is outdated and out-of-the-box support may seem bare-bones, but it has a whopping 1,800+ community-driven plugins available, so you can customize the tool the way you want, target it to any repository, and work with any programming language.Best fit for: Everybody. Jenkins is a Toyota Corolla of CI servers.
TeamCity: Proprietary Server with Great Out-of-the-box Support
TeamCity is a CI server by JetBrains. This is proprietary software that unlike Jenkins offers a lot of quality-of-life integrations right away. If you’re tracking issues in YouTrack, Jira, Bugzilla, Azure DevOps, or GitHub, TeamCity has native integrations with them to link issues in your bug-tracking software with your builds. It also connects with Docker and GitHub. The missing integrations are available in community plugins, the number of which hovers around 400. It falls short in comparison to Jenkins, but do you really need that many out of the box?On top of that, TeamCity comes with a number of build runners, integration middleware, that natively work with such build tools as Gradle, Maven, NAnt, MSBuild, and more. And the UI is much prettier than that of Jenkins.
This all comes at a cost. You can host the server yourself or get a cloud version, but either way, the number of agents and configurations is limited, unless you choose the Enterprise version.
Best fit for: JetBrains users and those who crave quality-of-life service.
Travis CI: 30 Languages Supported and Free for Open Source Tests
Travis is another popular CI tool that boasts a lot of integrations and support right out of the box. For instance, its native and community-driven tools allow you to build code across 30 programming languages without a complicated setup. Many languages have more than one build tool available. Travis connects with such version control systems as GitHub, Assembla, GitLab, and Bitbucket. The server is proprietary with tiered pricing, but running tests on open source projects is free!Best fit for: Those looking for out-of-the-box support of many languages and contributors to open source.
Bamboo: CI Server by and for Atlassian Ecosystem
Bamboo is an Atlassian product for those who are into their ecosystem. If you move tickets in Jira, document in Confluence, and version-control in Bitbucket...well, there’s no getting away from it. Bamboo native integrations cover Jira to check build results and link them to issues, Hipchat to get messages about builds, Bitbucket to work with VCS, Confluence to publish build charts, Fisheye to visualize code changes, and more. But even outside of the Atlassian ecosystem, Bamboo is generally a solid tool with about 200 community plugins.Best fit for: Those who belong to the ecosystem. Atlassian is just like Apple, but for engineers.
CircleCI: Rich Integration Set and Shareable Configurations
CircleCI also has a great set of integrations, which include build tools (Gradle, Maven, and more), cloud platforms, containers (Docker, AWS), Jira, deployment servers and serverless options (AWS SAM). The server also has a pool of their own, partner, and community-driven packages of configurations, which are called Orbs. The idea here is that you can use specific build configuration with its jobs and build tools to share it across different projects in your organization. There are a lot of orbs available out of the box, but you can configure custom ones as well. Orbs help standardize engineering processes and quickly set up your CI pipeline for a new application the way you’ve done this before.CircleCI also supports the open source community allowing for about $3k worth of open source builds on Linux and less than $200 worth of macOS builds.
The gripes around CircleCI vary, but most users complain about unintuitive UI and workflows. The learning curve isn’t gentle.
Best fit for: Those who need a lot of integrations and aren’t afraid of unintuitive workflows. Linux open source contributors are also welcome.
How to approach build automation and choose your CI server
Before you move on to examine tools and options further, let’s name key considerations.Write tests early. The whole point of running CI/CD pipelines and DevOps in general is to automate existing processes, including tests. Before you adopt a CI tool make sure that you have an underlying culture of covering code with tests. The best case scenario is that your user stories are written with automated tests in parallel. This will lay the groundwork for the further automation.
Consider your existing ecosystem. Different tools cater to different surrounding ecosystems. For example, GitLab also has their CI/CD solution. The same is for Azure cloud, suggesting a combined Azure DevOps platform. Start with what you’re using now and then explore alternatives.
Azure DevOps workflow, Source: Azure
Consider the background of people who set up CI. Who is going to manage your pipeline? For instance, operations and admin staff would be more comfortable with yaml configuration files used in CircleCI and Travis CI. TeamCity, on the other hand, has a dedicated UI, XML, and Kotlin support for configuration, which caters more to quality assurance specialists and engineers.Don’t forget about metrics and reporting. How do you report and track issues? If you use Jira, it’s great to have working integration with it in your CI server. If you speak the language of code coverage, build duration, and success rates, consider how you’re going to source these metrics from CI tools.
Dive deeper into supported languages, build, and test tools. The majority of CI server software already has some build and test tools support for various languages and frameworks. If your tools aren’t available right away most likely there’s a way to integrate that. Spend some time understanding which tools and packages your engineers already have. Estimate the effort to bring them onboard and connect with your build configuration inside a CI server.