Blog

How to Add BOSH Support to a Custom Cloud (Part 1): BOSH Components

Alexander Lomov

Part 1 | Part 2

how-to-add-bosh-support-to-a-custom-cloud-smallThe Cloud Foundry PaaS was designed to provide cross-cloud portability and compatibility. BOSH is the official orchestration and deployment tool for CF that makes these features possible. Currently, there is a set of cloud platforms that are able to work with BOSH, but it can be extended to work with clouds that are not on this list.

In this blog series, we’ll go through all the steps necessary to add BOSH support for a new cloud: from CPI implementation to generating a stemcell.
 

The process in a nutshell

In the recent years, the BOSH team have been working on an external CPI mechanism that will make adding BOSH support to new IaaS systems quicker. Still, implementing the full stack of solutions to accomplish this is rather difficult. The BOSH project is evolving very rapidly, so it is hard to find up-to-date information about it. There is also no explicit description of the full process for creating a BOSH CPI in available publications.

Enabling BOSH support for a new cloud is a large topic, so I’ve divided it into three parts:

  1. Understanding BOSH Essentials. This part covers some necessary background information on BOSH, as well as the requirements to your cloud infrastructure. You can skip it, if you already understand BOSH releases, CPI, stemcells, and MicroBOSH.
  2. Creating an External BOSH CPI. This blog post will describe how to implement a BOSH CPI in the most modern way and say a few words about the new way to bootstrap your cloud with MicroBOSH using the bosh-init (bosh-micro-cli) project.
  3. Building a Stemcell for a New IaaS. Currently, building stemcells for new infrastructures is not an easy task due to lots of reasons. This post will help you to cope with the issues that you are likely to encounter.

So, let’s get to know the BOSH components better. Here, I’m using the BOSH version 1.2922.0.
 

BOSH components and relations between them

If you are a beginner, I strongly recommend that you look through this documentation page and get familiar with the BOSH components before you start. In brief:

A BOSH release is a package that defines how software is compiled and how it should be run. So, it determines how sources are compiled, how templates are rendered, etc.

A job is an execution unit for BOSH. In fact, a job is a virtual machine where software from BOSH releases is executed. Jobs can contain content from several different releases.

BOSH CPI stands for BOSH Cloud Provider Interface. It is a programmatic wrapper between BOSH and the API of a specific IaaS. A BOSH CPI implements a list of methods defined by the Cloud class, such as create_vm, create_disk, attach_disk, etc. Originally, the CPI was implemented as a Ruby gem and a built-in BOSH project, but an external CPI mechanism has recently become available.

A stemcell is another cloud-specific BOSH component. In BOSH terminology, a stemcell is a cloud- or hypervisor-specific image that contains a pre-installed BOSH agent with associated packages, folder structure, groups, and users. These support the operation of BOSH inside an instance. In fact, every instance performing a BOSH job is created from a stemcell image. I will take a look at stemcell internals in the upcoming articles.

MicroBOSH is an official way to bootstrap a cloud with a BOSH installation limited to one instance. The term MicroBOSH is often used to refer to a one-instance deployment of BOSH. The main idea behind deploying MicroBOSH is to use BOSH itself to install BOSH components (more detail will be provided below).

The following diagram shows the relations between all these parts:

how-to-add-bosh-support-to-a-custom-cloud
 
So, if you decided to run BOSH on a new cloud, you will need to implement two things: a BOSH CPI and a stemcell builder. The CPI uses stemcells in its methods, such as create_stemcell and create_vm. The stemcell contains a MicroBOSH release with a CPI for the target cloud (the reason is described below). This cross-dependency causes lots of issues during development. In my upcoming posts, I am going to explain how to solve these issues.
 

Understanding MicroBOSH

Since MicroBOSH is used to install BOSH, you need to know how MicroBOSH works before you start implementing the CPI and building a stemcell for your cloud.

The standard MicroBOSH deployer is implemented as a BOSH CLI plugin. You can read about BOSH CLI plugins and how they work in my previous blog post. Almost the entire process of deployment is declared in the create method of the InstanceManager class. This process requires a MicroBOSH release to be built into the stemcell.

Let’s see how MicroBOSH works in case of OpenStack:

  1. First of all, you need to specify the stemcell and the manifest—the way it is described in documentation.
  2. When a user runs bosh micro deploy, this triggers the create_deployment method of InstanceManager and then the create method is called.
  3. Inside the create method, the create_stemcell CPI method is executed:
    • It takes an image file as a parameter and unpacks it to the root.img file located in a temp folder.
    • Using the glance service of the Fog library, it creates an image from the root.img file.
    • After the image has been created, the image ID is returned to BOSH.

  4. Then, the create_vm CPI is called. This method creates a VM using the image ID that was returned after the create_stemcell invocation.
  5. After this step, BOSH waits for the agent to be run on the VM.
  6. At this stage, the BOSH agent on the new VM gets a request to build a MicroBOSH release that is stored in the local file system of the VM. Instructions (or specs) for building a MicroBOSH release are stored in a stemcell file. They are passed to the agent by the BOSH Micro CLI plug-in.
  7. After the agent is done, MicroBOSH waits for the BOSH Director to finish his jobs, and the deployment is over.

The concept of MicroBOSH deployment is not so straightforward, but it’s not too hard to see how it works. Anyway, things are going to change soon with the new bosh-micro-cli tool that will be released in near future. You can find more about bosh-micro-cli in this conversation.
 

Issues to solve beforehand

After you get an understanding of BOSH basics, you’ll probably want to take a look at your infrastructure and check if it has all the necessary features. I suppose, your cloud is ready to perform such basic functions as creating and running VMs. What else will we need? Dealing with infrastructures of our customers, I have met several typical issues related to clouds that want to add BOSH support.

First of all, some clouds have difficulties with creating new custom images. By “custom image” I mean a template for a bootable system disk that will become a VM. This is how we create stemcells. The stemcell is generated with a special builder, which is a part of the BOSH project. This is done to ensure that BOSH components will work in an environment with specific binary libraries. The way the stemcell builder is implemented implies that your cloud can use a file as a “custom image.” In case of OpenStack, this is the .qcow2 file format built using the qemu-img tool. vSphere uses the vmdk files built with the ovftool utility. If you can’t use the same workflow, you will need to implement “lightweight” stemcells. I’ll provide more details on this solution in the upcoming posts.

Second, BOSH saves data to persistent disks. This means the data is retained after an instance has been rebooted. It is possible to make BOSH work with clouds without persistent disks, but such a deployment will be unstable. For instance, you can take a look at this Twitter discussion about BOSH support for DigitalOcean.

There are also some optional features, e.g., sometimes it is useful for BOSH to write data to a metadata service. A metadata service helps BOSH to optimize the working process and use tags and names for instances. Such services are implemented in the most popular infrastructures, e.g., AWS, OpenStack, and GCE. This is an optional requirement for BOSH deployment. In fact, using a metadata service for sensitive data in public clouds can be dangerous. You can avoid it altogether by relying on a BOSH registry.

Blob storage is another nice-to-have feature for your cloud. This is where BOSH stores compiled packages used by jobs. The AWS CPI uses the S3 service as blob storage, while Swift is used in case of the OpenStack CPI. This doesn’t mean you cannot run BOSH without blob store support. There are different substitutions for it, e.g., the BOSH project has a simple built-in blob store server.
 

Next: an external BOSH CPI

By now, you should be familiar with the basic BOSH terminology and how BOSH interacts with different IaaS systems. In the upcoming posts, I will go through the steps for creating an external BOSH CPI module and building a stemcell for a new cloud.


About the author: Alexander Lomov is a Cloud Foundry Engineer at Altoros. With extensive Ruby experience, Alexander is a fan of the open source movement, having contributed to Fog, RefineryCMS, Netzke, simple_form, and other projects. His professional interests include AWS, MySQL, PostgreSQL, Cassandra, MongoDB, C++, jQuery, JavaScript, and much more.


Subscribe to our blog for the next parts of this series or follow @altoros.

Get new posts right in your inbox!

4 Comments
  • Good article. BTW, bosh-micro-cli repo now renamed to bosh-init – https://github.com/cloudfoundry/bosh-init

    • Alexandr Lomov

      Thank you, Dr. Nic. This is a good note. Yes, I worked with bosh-micro-cli when I started to write these articles. I corrected it in the next.

  • Marco Völz

    Dear Alexander, thanks for the great writeup! This is very useful to understand the internals of BOSH and how to get started with your own CPI.

    I’m not sure about your account of the BOSH registry and metadata service. Depending on your IaaS, the metadata service might be a necessary bootstrapping mechanism, as the location of your registry and method of device-adress-scheme (e.g. virtio) are configured in there. You might be able to hardcode these things in your own CPI, I don’t know what would be the solution here.

    If I understood Dmitriy correctly, the registry might not be around forever – in fact it is just a poor means to achieve a way to re-configure a running agent, as far as I know.

    • Alexandr Lomov

      Hey, Marco.

      Thank you for interesting answer. Actually you can set up your device-adress-scheme and metadata mechanism into settings of bosh-agent during stemcell creation. bosh-agent basically works with cloud-init to get more information about itself, but there is a way to say agent to load data from a files. You can see implementation of agent’s FileMetadataService here https://github.com/cloudfoundry/bosh-agent/blob/master/infrastructure/file_metadata_service.go (in fact this is implementation of client for the case when you specify “File” as type of settings source). In fact you can upload file to the server with the agent and it will be treated as a metadata.

      Talking about registry I have to say there are plans to get rid of it, but still almost every CPI I saw was using registry. Also registry is used inside public clouds to store sensitive data. I will need to cover this two topics in next articles.

      Thank you one more time for your comment. You are welcome to leave more if there any further questions.

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!