Blog

An Outstanding Cloud Foundry Feature: App Placement and Host Affinity

Aliaksandr Prysmakou

an-outstanding-cloud-foundry-feature-app-placement-and-host-affinity-v2

Out of the box, Cloud Foundry doesn’t give us the ability to control application placement or to specify a target host(s) for an application. This may be critical when one needs to separate apps between different environments or host types—or for any other type of app segregation. The good news is that this can be resolved very easily.

 

Why bother?

Scenarios where this ability may be needed:

  • physical separation of applications between environments in a single Cloud Foundry deployment (e.g., development / staging / production)
  • segregation of applications by the host’s hardware (slow_but_cheap/fast_but_expensive/with_gpu)
  • placing together two microservices that have a huge amount of communication traffic and will benefit from lower latencies
  • a use case with only one app instance per host (e.g., a crypto that encrypts traffic for the whole host)
  • well, actually, for any type of segregation

Ok, but we already have Availability Zones (AZ)!

It is a different thing. Cloud Foundry tries to balance (using internal algorithms) instances of an application between AZs. There is no way to specify a target AZ for an application or an instance of an application. “Host affinity” or “placement constraints” are needed to control application placement.

For instance, a reference architecture of your deployment may look like this:

an-outstanding-cloud-foundry-feature-app-placement-and-host-affinity

 

Solution (well, a workaround)

In brief, it is possible to control application placement by defining stacks with custom names in the Cloud Foundry deployment manifest. There is no need to actually create them and no release modification is needed.

Cloud Foundry uses the concept of stacks (prebuilt root file systems) to support specific operation systems.

Examples:

  • cflinuxfs2
  • windows2012R2

However, nothing stops us from listing more stacks, even without creating modifications to them:

  • cflinuxfs2_dev
  • cflinuxfs2_stg
  • cflinuxfs2_prd
  • name_of_your_choice

 

Instructions for Diego

In my example, I define a “production” affinity group of hosts so that every application for production (and only for production) will run on the specific group of hosts. All you need is to modify your Cloud Foundry deployment manifest. No need to create new stacks. No need to modify a CF release.

Just list your stacks in the Cloud Controller properties:

properties:
...
  cc:
    stacks:
      - name: "cflinuxfs2"
        description: "Cloud Foundry Linux-based filesystem"
      - name: "cflinuxfs2_prd"
        description: "Cloud Foundry Linux-based filesystem (PRODUCTION)"

Define your “affinity group” (REP properties):

- instances: 10
  name: cell_z1_prd
  networks:
  - name: diego_prd1
  properties:
    diego:
      rep:
        stack: cflinuxfs2_prd
        preloaded_rootfses: 
          - "cflinuxfs2_prd:/var/vcap/packages/rootfs_cflinuxfs2/rootfs"
        zone: z1

Modify Diego properties for the NSYNC and STAGER components:

properties:
  diego:
...
    nsync:
      lifecycle_bundles:
        - "buildpack/cflinuxfs2:buildpack_app_lifecycle/buildpack_app_lifecycle.tgz"
        - "buildpack/cflinuxfs2_prd:buildpack_app_lifecycle/buildpack_app_lifecycle.tgz"
        - "buildpack/windows2012R2:windows_app_lifecycle/windows_app_lifecycle.tgz"
        - "docker:docker_app_lifecycle/docker_app_lifecycle.tgz"

...
    stager:
      lifecycle_bundles:
        - "buildpack/cflinuxfs2:buildpack_app_lifecycle/buildpack_app_lifecycle.tgz"
        - "buildpack/cflinuxfs2_prd:buildpack_app_lifecycle/buildpack_app_lifecycle.tgz"
        - "buildpack/windows2012R2:windows_app_lifecycle/windows_app_lifecycle.tgz"
        - "docker:docker_app_lifecycle/docker_app_lifecycle.tgz"

 

Instructions for DEA

The same as with Diego, all you need is to modify your Cloud Foundry deployment manifest.

List your stacks in the Cloud Controller properties:

properties:
...
  cc:
    stacks:
      - name: "cflinuxfs2"
        description: "Cloud Foundry Linux-based filesystem"
      - name: "cflinuxfs2_prd"
        description: "Cloud Foundry Linux-based filesystem (PRODUCTION)"

Define your affinity group (DEA properties):

- instances: 10
  name: runner_z1_prd
  networks:
  - name: cf1
    static_ips: null
  properties:
    consul:
      agent:
        services:
          dea:
            check:
              interval: 5m
              name: dns_health_check
              script: /var/vcap/jobs/dea_next/bin/dns_health_check
              status: passing
    dea_next:
      zone: z1
      stacks:
        - name: "cflinuxfs2_prd"
          package_path: "/var/vcap/packages/rootfs_cflinuxfs2/rootfs"

 

Example application manifest

To target specific affinity group (in my example, “production”) just specify the appropriate stack in your application manifest:

---
applications:
- name: hello-spring-cloud
  memory: 512M
  instances: 1
  host: hello-spring-cloud-${random-word}
  path: target/hello-spring-cloud-0.0.1-SNAPSHOT.jar
  buildpack: java_buildpack
  stack: cflinuxfs2_prd
  env:
    SPRING_PROFILES_DEFAULT: cloud

 

Conclusion

So, as you can see, it is pretty easy to define affinity groups in Cloud Foundry and to control application placement. It would be great to see this as a native feature in Cloud Foundry, but this workaround also works well.

I’d also like to see the ability to specify placement constraints in the application manifest. Just like this:

---
constraints:
- production
- with_gpu

Pros:

  • No need to build your own custom stack
  • No need to modify a CF release
  • No need to modify your applications
  • Works for DEA and Diego cells

Cons:

  • No way to request a list of constraints
  • May require buildpack customization (The Java buildpack works well, but I’ve experienced some limitations for the Python buildpack.)

In addition, to separate app traffic, it is also worth considering using different load balancers / routers for different affinity groups (at the DNS level). See the image above for possible separation of app traffic. Please share your thoughts if you have anything else to add.


Many thanks to Sean Keery (Pivotal) and Duncan Winn (Pivotal) for inspiration!


About the author

Aliaksandr Prysmakou is a Cloud Foundry DevOps Engineer at Altoros. He is an expert in cloud automation and virtualization. As a member of a joint team formed by Canonical, Pivotal, and Altoros, Alex worked on automating Cloud Foundry deployment with the Juju orchestration platform. He also contributed to extending BOSH to support new cloud providers. Previously, Alex designed templates to automate management of Microsoft solutions on the RightScale platform and provided fault tolerance for Microsoft SQL Server.


Related reading:


To stay tuned with the latest updates, subscribe to our blog or follow @altoros.

Get new posts right in your inbox!

4 Comments
  • Sean-Pivotal

    Awesome

  • C. A. Ferraris

    A shout-out would have been appreciated 🙂 https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/thread/WUGWFUXL6ANTWIILYZ3CNPLNLCV2W7F2/

    • Aliaksandr Prysmakou

      To be honest, I thought everybody knows this trick. I started to work on the blog-post when folks asked about details.

  • Alexandre Vasseur

    This Rakuten testimony at CF summit 2016 describes it very well also – and their video is available on youtube as well

    http://schd.ws/hosted_files/cfsummit2016/28/5%20Year%20of%20Cloud%20Foundry%20at%20Rakuten%20.pdf

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!