8 Pro Tips for Using Concourse CI with Cloud Foundry
There are at least a dozen continuous integration solutions out there, but the majority of them are challenging to learn, scale, and debug. Most importantly, few of them were designed with pipelines in mind. Not able to find a tool that could handle something as massive and complex as Cloud Foundry, Pivotal decided to write their own. This resulted in Concourse CI, a user-friendly continuous delivery technology that focuses on pipelines, is easy to scale, runs anywhere, and works with anything.
From this post, you will learn how Concourse CI differs from other similar products, how it works, and how it can help you to “nitrocharge” your own projects.
Why Concourse CI is different
Here are some of the features that really make it stand out:
- Pipelines are first-class citizens. Concourse CI is the first open source tool built around pipelines, like Atlassian Bamboo. Unlike it, Concourse has no proprietary license and does not lock you on any technologies.
- Containers and microservices. Everything runs in ephemeral, reproducible containers. Horizontal scaling is a wheeze. Each part of the pipeline can be scaled independently.
- Everything-as-Code. Deployments, builds, and anything can be automated, scripted, and then re-created without wizards or manual configuration.
- Infrastructure agnostic. Concourse CI is deployed with BOSH and can run on any cloud with BOSH support (AWS, GCE, vSphere, etc.), as well as on bare metal.
- Written in Golang. It is possible to test software on various platforms, running workers on Linux, Windows, or MacOS (like it is done for Fly CLI).
- Built for Cloud Foundry with love. Concourse CI was created to streamline development of the Cloud Foundry PaaS and can be readily integrated with BOSH.
- Other perks. Concourse is packed with features that facilitate building and debugging CD pipelines (e.g., pipeline visualization, user-friendly text formatting, etc.).
For a detailed introduction with demos, watch this Cloud Foundry Summit talk by Alex Suraci, one of Concourse CI’s main contributors.
Main concepts and architecture
Concourse CI strives for simplicity in everything. There are only three core concepts.
- Resources: anything one can publish to or pull from and anything that can be versioned (e.g., GitHub, AWS S3 buckets, Docker images, BOSH deployments / releases / stemcells, etc.). Each resource is a container with scripts that can interact with external objects, such as S3.
- Tasks: instructions on how something should be run (container properties, dependencies, etc.)
- Jobs: these combine resources and tasks, which results in something being done.
Resources, tasks, and jobs make up pipelines where each part is relatively independent and can be re-used, even for another project. Visualization makes it easy to follow the stages that the artifacts go through inside the pipeline.
A distinct feature is that both resources and jobs are run in containers. Resources may be written in any language, which makes it simpler to integrate Concourse CI with other tools. As a bonus, you can quickly “SSH” into any container for debugging with the Fly CLI command. This beauty is brought to Concourse by Garden, a re-implementation of Cloud Foundry’s container manager—Warden—in Golang.
The architecture of Concourse CI is very straightforward, with cool aircraft-inspired naming (see the diagram above). At the heart of it, lies Air Traffic Control (ATC). ATC provides a REST API and Web GUI, processes all logic, and manages all workflows. It works closely with Transport Security Agency (TSA), which exposes an interface to workers, allowing them to register with ATC.
Workers consist of several components, the main one being Garden. To provide Garden containers with a remote layered file system, the baggageclaim service is used. Workers are also serviced by groundcrew, which registers them with ATC at certain intervals, advertises supported resource types, and cleans up ephemeral containers on shutdown.
A detailed description of the architecture can be found in Concourse docs.
Getting started with Concourse CI
Concourse CI can be deployed with BOSH, via standalone binaries, or in a Vagrant box. Each way produces a fully functional deployment, the only difference being scalability and how fast you can get it working. Using a Vagrant box and the concourse-lite project (similar to bosh-lite), you can get Concourse CI on AWS or a local machine in minutes. Updates for Vagrant come very frequently, thanks to the Concourse CI team. For more on how to use Concourse, read this tutorial.
Once you have a working Concourse CI deployment, it is trivial to build CD for projects of any size. Still, there are some things that you should keep in mind to avoid running into issues down the road. Here is a selection of tips based on our experience of working with various Concourse CI clusters. They will help to get your CD right from the first try:
- Follow best practices. Every week, a question about best practices is raised in the Concourse Slack channel and Alex Suraci answers it, using examples from pipelines for concourse CI. This is where you can learn that the common practice is to have a ci folder in the project repo. The ci repo should contain folders called tasks, docker, and pipelines for tasks, Docker files, and pipelines respectively. If there is only one pipeline, it is better to save it to a pipeline.yml file directly under the ci folder. It is also preferable to have a separate directory for scripts and assets. The screenshot below shows an example of the approximate structure, which you can generate, for instance, with our concourse-helpers gem. The gem will also create your pipeline to test the framework you are using.
For more ideas on how to arrange CI pipelines, read the sources of Diego, buildpacks, and Garden Linux.
- Use CPU-intensive VMs for Concourse CI workers. Since running a lot of containers on one VM is a CPU-intensive task, it is a good idea to have fewer workers and a larger number of CPUs per machine (see this discussion). If you work with physical machines, it is best to have one worker per machine.
- Use notifications for dependency updates. Keeping up with changes to dependencies is vital for any cloud project. E.g., the Cloud Foundry Buildpacks Slack channel uses this Slack bot to post notifications every time a buildpack dependency is updated. Another example is Altoros’s Jumpbox Ansible role, which is updated automatically whenever a dependency gets updated.
- Don’t forget about security. Concourse CI’s TSA components address most security threats. TSA even allows users to connect ATC to workers via the Internet. The only thing to care about is authentication. Currently, there are two main types of it: basic HTTP and GitHub-based.
- Expose the Concourse CI cluster. In most cases, this will require a load balancer, like HAProxy. Alternatively, you can install this router-registrar project with Concourse CI deployed with BOSH to conveniently expose the cluster through Cloud Foundry.
- Code your logic. Concourse works as a service and can be customized or extended with other services, such as the go-concourse client for Golang.
- Save your logs. Getting a log aggregator to store and organize logs is common advice for anyone maintaining a service that generates a lot of activity. There are many tools that can help with that. I would recommend the ELK stack BOSH release, which can be found here. We are planning to publish a blog post about this wonderful tool in the coming months.
- Collaborate. Remember that you can always find engineers ready to help and answer questions in the Concourse Slack channel.
Concourse CI is awesome for Cloud Foundry or any large cloud-native project that uses microservices. It is written entirely in Go and is available as standalone binaries that run on any platform / architecture with Go support: Windows, Linux, or OS X, as well as IBM POWER8, ARM, etc. Developing resources in Go means we can reduce pipeline execution time, using lightweight images for containers (e.g., Alpine or BusyBox). Alternatively, apps and dependencies can be packaged as Docker images. Last, but not least, Concourse CI supports different types of authentication, including Vault.
Concourse CI is already employed by dozens of companies, Altoros among them. Our team is also working on some new functionality that will make Concourse CI even better. We will keep you updated on the progress.
About the authors:
Alexander Lomov is a Cloud Foundry Engineer at Altoros. With extensive experience in Ruby, Go, and Python, he was involved in development of BOSH CPIs and other Cloud Foundry-related projects for Canonical, IBM, and other companies. Alexander is a frequent speaker at various events/meetups, mostly sharing his experience with Cloud Foundry. You may also know him as the author of several blog posts about Cloud Foundry internals.
Volha Kurylionak is Technical Writer at Altoros, a Cloud Foundry activist, a technical blogger with 4+ of experience, and a big fan of open source.
To stay tuned with the latest updates, subscribe to our blog or follow @altoros.