Blog

Are Diego and Docker Really Good Friends?

Lev Berman

cloud-foundry-docker

Support for Docker is one of the main advantages of Cloud Foundry Diego, but how far does this compatibility go? Is it possible to push an arbitrary image from the Docker Hub to Diego and, if not, what are the constraints? What’s going to change in the future? Finally, why should anyone want to use Diego with Docker at all?

In this post, I’ll answer all of these questions, as well as show how to customize and push the official Redis image and demonstrate how to turn it into a simple service to be consumed by other apps.

 

Technical prerequisites

I’m going to examine the features of the latest (as of the time this post was written) Cloud Foundry release (v207) and the corresponding version of Diego (release v0.1099.0). The functionality that I will describe has been tested on AWS deployments of Cloud Foundry and Diego.

Diego has to be configured to use the Docker backend (the diego_docker manifest property has to be set to true). In order to push Docker applications, the Diego plug-in for the CF CLI has to be installed.

 

How Diego supports Docker

When I talk about Docker support, I mean that some Docker image, which Docker can create a container from and run processes in, is consumed by Diego to create a container, in which the same processes can run seamlessly.

To provide such support, Diego uses Garden-Linux that constructs Linux containers and mounts Docker images to them. While not directly using Docker to create containers, Diego employs some of the libraries Docker is built with. With them in place, Garden-Linux extracts the necessary metadata (mainly the commands to be launched) from a Docker image, mounts all the image layers to the container it has created, and executes the starting commands along with the Diego health check script when the container is launched. Here, you can find more details about the Docker support in Diego, as well as some notes about the compatibility issues and work in progress.

 

The main incompatibilities

The two main pitfalls that may prevent some Docker images from running on Diego are the health check script indispensably checking the 8080 port and container processes running as the vcap user with limited permissions and ownership rights.

This is not a big deal if you run a regular user application that normally doesn’t need to have root privileges or use files in system directories that may require some permissions. In that case, you can usually configure the port to fit the external requirements. The “port issue” does not apply to the apps that do not need routes either. Issues arise if you are working with applications that are actually services: Web servers, databases, message queues, etc.

 

Available workarounds

The simplest solution for the issues described above is customizing your Docker images so that they match Diego’s constraints.

Consider the official Redis image as an example. If we look through its Dockerfile and the starting script, we’ll see that the redis user is created and then used for starting the service. This won’t work in Diego, because the vcap user doesn’t have the privilege to set the user ID. What we can do here is run Redis as the vcap user.

Still, we may need to change user permissions and/or ownership rights. The latter is impossible right now, but there is a workaround to resolve the former. Permissions can be set when a Docker image is created. If we set the proper bits for the other users, they will be applied to the vcap user later when the command will be executed:

RUN chmod 777 /data

Running as the vcap user with proper permissions granted should be enough for most services (Redis, MongoDB, etc.), because they rarely rely on ownership of files. In case of the Redis image though, it is sufficient to simply remove the corresponding user-manipulation directives.

And what about the port? That’s easy. In case of Redis, we just need to provide the –port argument to the redis-server command.

The modified Redis image with the fixes described above can be found in the Altoros’s GitHub repo. Redis can be installed from our repo on Docker Hub. The installation process is as simple as:

cf docker-push redis altoros/redis

 

Turning a Docker app into a CF service

We have deployed Redis from the Docker image in almost the same way any regular app can be deployed. Why may we need such a service and how can we use it?

One use case I can see here is a service in a private PaaS that can be easily created either by an operator or by a developer to be consumed by a set of apps deployed to this platform. There are three main reasons why you may want this kind of service:

  • The simplicity of service deployment
  • The convenience of further service development (you can debug it locally with Docker)
  • The advantages provided by Diego, including, but not limited to, health checking, monitoring, log aggregation, and scaling (in case of stateless HTTP services)

So, how do you set it up? One thing Cloud Foundry helps us to do is connect services with apps via user-provided service instances. We can create an instance with the CF CLI, giving it the necessary credentials:

cf create-user-provided-service redis -p ‘{“host”:”<host>”,”port”:”<port>”,”db”:”<db>”}’

How do we know the exact host and port? Since every Diego application is actually a so-called Long Running Process (LRP), we can query the vital information via the Receptor HTTP API:

curl http://receptor.<CF_DOMAIN>/v1/actual_lrps

As a response, we will receive an array of JSON objects, describing the currently running LRPs. The object we need can be found by the process_guid field that is the corresponding app’s GUID (cf app redis –guid). The pieces of information we are looking for are the address field that corresponds to the host parameter and the container_port field that corresponds to the service port. The db parameter is just an index of the database that Redis will use.

There is one more step here: You have to configure a Cloud Foundry security group to allow incoming TCP traffic for the service. For more detailed instructions, refer to the README file of cf-mysql-release.

Some important things to note here:

  • The application route is not of any use to Redis, because Router in Cloud Foundry has no support for TCP traffic. Sure, if you deploy the HTTP service, the route just works and there is no need to investigate the host and port.
  • It is up to you which credentials to provide. This depends on the applications that consume the service (e.g., in case of Redis, you can additionally set the password or omit the db parameter, so that it defaults to “0”).
  • Each time you start an app, Diego assigns a new port to it. After a restart, the service needs to be updated (cf update-user-provided-service) and its consumers need to re-bind it and re-stage (yeah, that’s lame).
  • A user-provided service instance exists only inside the space it was created in.
  • There is currently no way to get a user-provided service instance with tags. Therefore, your application needs to search for the service by name. This results in some constraints related to binding multiple instances of the same service to multiple instances of one application.
  • Containers that require persistent data is one of the anti-goals of the Diego team. Therefore, this approach is not suitable for databases and other services that are used for persistent data storage.

I’ve modified a popular cf-redis-example-app from Pivotal to work with the user-provided Redis instance. You can get it from GitHub. Note that if you want to push the app to the Diego backend, the external network access has to be explicitly enabled to consume the Redis service.

 

Summing it up (for Ops and Devs)

To conclude, below is what the above described process looks like from an operator’s and developer’s perspectives.

The operator way:

cf docker-push redis altoros/redis
cf app redis --guid  # we’ll use this in the next command
curl http://receptor.<CF_DOMAIN>/v1/actual_lrps/  # search for the host and port
cf cups redis -p ‘{“host”:”<HOST>”,”port”:”<PORT>”,”db”:”0”}’

# create a security group to allow the incoming tcp traffic

The developer way:

git clone https://github.com/Altoros/cf-redis-example-app.git
cd cf-redis-example-app
cf push
cf bind-service redis-example-app redis
cf restage redis-example-app
# try it:
curl -X PUT -d “data=value” <APP_ROUTE>/testkey

The service developer way:

git clone https://github.com/Altoros/redis.git
cd redis/3.0
# do some modifications
docker build -t altoros/redis .
docker run altoros/redis  # local debugging
docker push altoros/redis
cf restage redis

 

Future plans for Diego, Docker, and Lattice

Diego is currently in active development. Docker support is being improved, as well. Soon, it will be possible to run processes as an arbitrary user.

As I talk about Diego and Docker, I can’t simply skip Lattice—a project that is now a better friend of Docker than Cloud Foundry Diego. Lattice is a platform that offers a cluster scheduler, HTTP load balancing, log aggregation, and health management for containerized workloads. Its container runtime is based on Diego, but some additional steps have been made towards Docker support. In the documentation, there are numerous examples of official Docker images that can be run on Lattice. In addition, Lattice can be configured to communicate with a private Docker registry.

For more details about the future of Diego and Docker, read the Google doc shared by the Diego development team.

 

About the author

Lev Berman is a Go developer and Cloud Foundry engineer at Altoros. He has 3+ years of experience in software development, including building cloud-native applications. Working both as a developer and DevOps engineer, Lev currently focuses on BOSH, Cloud Foundry/Diego, and containerization technologies, such as Docker. He also holds an MS degree from the Moscow Institute of Physics and Technology.




Related video: Microservices, Docker Containers, Cloud Foundry, and Diego


Subscribe to our blog for the latest updates or follow @altoros.

Get new posts right in your inbox!

24 Comments
  • Victor Fonseca

    How do I install receptor api on cf. I’m using bosh-lite cloudfoundry deploy.

  • Victor Fonseca

    How do I install receptor api on cf. I’m using bosh-lite cloudfoundry deploy.

    • Lev Berman

      Hi Victor,

      You have to install Diego alongside your Cloud Foundry deployment. The Diego release repo contains a comprehensive step-by-step guide to accomplishing this with bosh-lite – https://github.com/cloudfoundry-incubator/diego-release#deploying-diego-to-bosh-lite.

      I hope this helps.

      • Victor Fonseca

        Already install diego. But, when I try to curl http://receptor.bosh-lite.com come back with a 404 error. But I can curl api, ssh, uaa and others.

        • Lev Berman

          Is bosh-lite.com your domain? Anyway, 404 seems to be a valid response. Could you, please, check whether /v1/actual_lrps responds with a JSON array (possibly empty)?

          • Victor Fonseca

            cf curl /v1/actual_lrps -X GET

            {

            “code”: 10000,

            “description”: “Unknown request”,

            “error_code”: “CF-NotFound”

            }

            That’s what I get.

          • Victor Fonseca

            If I use this, I get port and ip of container running is the same thing from actual_lrps?

            cf curl /v2/apps/d428d19e-cf63-4407-ab09-098b752ed2d7/stats -X GET

            {

            “0”: {

            “state”: “RUNNING”,

            “stats”: {

            “name”: “spring-music”,

            “uris”: [

            “spring-music.bosh-lite.com”

            ],

            “host”: “10.244.16.138”,

            “port”: 60000,

            “uptime”: 1329,

            “mem_quota”: 1073741824,

            “disk_quota”: 1073741824,

            “fds_quota”: 16384,

            “usage”: {

            “time”: “2015-11-29T15:04:31.256709611Z”,

            “cpu”: 0.005361167262428518,

            “mem”: 521105408,

            “disk”: 194052096

            }

            }

            }

            }

          • Lev Berman

            cf curl talks to Cloud Controller, not Receptor. I’ve meant `curl http://receptor./v1/actual_lrps/`.

          • Victor Fonseca

            Same 404, curl receptor.bosh-lite.com/v1/actual_lrps/

            404 Not Found: Requested route (‘receptor.bosh-lite.com’) does not exist.

          • Lev Berman

            What does `bosh vms` show?

          • Victor Fonseca

            eployment `cf-services-contrib’

            Director task 682

            Task 682 done

            +———————-+———+—————+————-+

            | Job/index | State | Resource Pool | IPs |

            +———————-+———+—————+————-+

            | mongodb_gateway/0 | running | gateway_z1 | 10.244.1.2 |

            | mongodb_node/0 | running | node_z1 | 10.244.1.82 |

            | postgresql_gateway/0 | running | gateway_z1 | 10.244.1.10 |

            | postgresql_node/0 | running | node_z1 | 10.244.1.90 |

            | rabbit_gateway/0 | running | gateway_z1 | 10.244.1.6 |

            | rabbit_node/0 | running | node_z1 | 10.244.1.86 |

            | redis_gateway/0 | running | gateway_z1 | 10.244.1.14 |

            | redis_node/0 | running | node_z1 | 10.244.1.94 |

            +———————-+———+—————+————-+

            VMs total: 8

            Deployment `cf-warden’

            Director task 683

            Task 683 done

            +————————————+———+—————+————–+

            | Job/index | State | Resource Pool | IPs |

            +————————————+———+—————+————–+

            | api_z1/0 | running | large_z1 | 10.244.0.134 |

            | consul_z1/0 | running | small_z1 | 10.244.0.54 |

            | doppler_z1/0 | running | medium_z1 | 10.244.0.142 |

            | etcd_z1/0 | running | medium_z1 | 10.244.0.42 |

            | ha_proxy_z1/0 | running | router_z1 | 10.244.0.34 |

            | hm9000_z1/0 | running | medium_z1 | 10.244.0.138 |

            | loggregator_trafficcontroller_z1/0 | running | small_z1 | 10.244.0.146 |

            | nats_z1/0 | running | medium_z1 | 10.244.0.6 |

            | postgres_z1/0 | running | medium_z1 | 10.244.0.30 |

            | router_z1/0 | running | router_z1 | 10.244.0.22 |

            | runner_z1/0 | running | runner_z1 | 10.244.0.26 |

            | uaa_z1/0 | running | medium_z1 | 10.244.0.130 |

            +————————————+———+—————+————–+

            VMs total: 12

            Deployment `cf-warden-diego’

            Director task 684

            Task 684 done

            +——————–+———+——————+—————+

            | Job/index | State | Resource Pool | IPs |

            +——————–+———+——————+—————+

            | access_z1/0 | running | access_z1 | 10.244.16.6 |

            | brain_z1/0 | running | brain_z1 | 10.244.16.134 |

            | cc_bridge_z1/0 | running | cc_bridge_z1 | 10.244.16.142 |

            | cell_z1/0 | running | cell_z1 | 10.244.16.138 |

            | database_z1/0 | running | database_z1 | 10.244.16.130 |

            | route_emitter_z1/0 | running | route_emitter_z1 | 10.244.16.146 |

            +——————–+———+——————+—————+

          • Lev Berman

            Sorry, I have started pointing you in the wrong direction. Receptor was removed from the project in 0.1432.0 – https://github.com/cloudfoundry-incubator/diego-release/releases/tag/0.1432.0, and you probably work with a more recent version. You should use Cloud Controller API to get the information about apps. Eg as you have already mentioned `cf curl /v2/apps/d428d19e-cf63-4407-ab09-098b752ed2d7/stats -X GET` is a perfect way to get to know the host and port of an app.

          • Victor Fonseca

            Thank’s Lev! You are the best. I try that yesterday, works fine, but I have some problems yet with Security Groups, but I will figure it out.

        • Victor Fonseca

          If I use cf curl /v2/apps/d428d19e-cf63-4407-ab09-098b752ed2d7/stats -X GET is the same?

          http://apidocs.cloudfoundry.org/runtime-passed/apps/get_detailed_stats_for_a_started_app.html

  • webdevcloud

    Lev,

    I am working on jboss/drools-workbench-showcase image.

    I pushed this image to cloudfoundry, the app is running successfully.

    cf push drools -o jboss/drools-workbench-showcase

    I can see the server running when launch the application.

    And when I launch the application drools.cf-example.domain.com/drools-wb, I get 502 error.

    My understanding is cf push -> pushes the docker image to CF -> like docker pull

    But how to run the docker image in cloud foundry.?

    • Andrei Krasnitski

      Hi webdevloud,
      You faced this issue because Diego allows you to run only cloud adapted applications, as a workaround i can suggest you to push “ollin/drools-wb-tomcat7” Docker image.
      I checked and can confirm that this image works with Diego and Drools Workbench web UI will be available on the following link http://drools.cf-example.domain.com/droolswb .

      • webdevcloud

        Thank you Andrei,
        I did try to pushing this image.
        I see Start app timed out.
        ———
        .
        .
        0 of 1 instances running, 1 starting
        FAILED
        Start app timeout

        TIP: Application must be listening on the right port. Instead of hard coding the
        port, use the $PORT environment variable.
        —————–

        But the application is running in CF, and logs has lot of error.

        Web UI -> cf.example.doamin.com/ -> I can see wildfly is running page
        cf.example.domain.com/droolswb -> 404 Error

        I am running on PCF 1.6.
        Any help on this?

        • Andrei Krasnitski

          I’ve repackaged the container for you. Please try pushing it again using this command: ‘cf push drools -o andreikrasnitski/cf-drools-wb-tomcat’. We have a local Pivotal Elastic Runtime v1.6.20 deployment and this image works with this PCF version without any issues. If your problem still persists, I suggest you try to upgrade your PCF installation.

          • webdevcloud

            Thank you Andrei,
            I have the app running without any log erros.
            But drools./droolwb is not accessible.
            My cloud team is planning to upgrade to PCF 1.7
            Thanks a lot.

          • webdevcloud

            Andrei,
            This is the error I see after starting the docker image in CF. ,
            I am working on PCF ER 1.6.15 with proxy enabled. The image andreikrasnitski/cf-drools-wb-tomcat workis good in PCF 1.6.20.
            Can you post the source code of the docker image also.

            ——————————-
            2016-05-19T09:24:59.91-0400 [APP/0] ERR May 19, 2016 1:24:59 PM org.apache.catalina.core.StandardContext startInternal

            2016-05-19T09:24:59.91-0400 [APP/0] ERR SEVERE: Error listenerStart

            2016-05-19T09:26:46.92-0400 [APP/0] ERR May 19, 2016 1:26:46 PM org.apache.catalina.util.SessionIdGeneratorBase createSecureRandom

            2016-05-19T09:26:46.92-0400 [APP/0] ERR INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [107,003] milliseconds.

            2016-05-19T09:26:46.92-0400 [APP/0] ERR May 19, 2016 1:26:46 PM org.apache.catalina.core.StandardContext startInternal

            2016-05-19T09:26:46.92-0400 [APP/0] ERR SEVERE: Context [/droolswb] startup failed due to previous errors

          • Andrei Krasnitski

            Hi!
            Please find the link below:
            https://github.com/Infra-Red/cf-drools-wb-tomcat.git

          • webdevcloud

            can I have a email contact. I am new to docker and CF.

  • Pola

    Hi Lev,

    I’m a CF developer and I need to deploy a R/RServe that can be used by other applications. I put my image in the public docker registry, and I’m following your article to make it available in in my cloud foundry landscape.

    1. I publish my docker image as an app, expose the port 8080;
    2. I obtain the IP/Port of the corresponding instance of my docker application (the port 8080 has been mapped to a host port);
    3. I create a user provided service from the docker instance IP/Port, bind it to another java application I have deployed

    Then I realize that from my java application, I cannot connect to the docker instance. Always the same error “Connection refused”. In fact, I got the same error with or without creating that user provided service.

    I notice that we can create a security group to allow TCP communications between services, I try to create one, and then I got an error that I’m not authorized to do so.

    So is this an obligation to create a security group to make available my user provided service? If that’s case I guess I’ll have no choice but ask for the operator to do the work. Otherwise, could you suggest what else could be wrong?

    thanks,
    Pola

    • Alexandr Lomov

      Hey, Pola!

      Things have changed since this article was written. Could you tell your CF version? Also what CF CLI do you use?

      My guess is that you use rather new release of CF, so you need to perform some changes on docker image that you push to CF runtime.

      First of all you should not expose port 8080 and you need app in your container to be run on the port that is determined by `$PORT` environment variable. This is done to allow multiple apps run on the same host with garden.

      The second thing is that an IP address that you get inside of your app is actually an IP address in the network created inside of garden host. Having multiple garden VMs will mean you’ll have several distinct networks and app will not reach each other. That’s why a common way to expose your app/service in CF is to use gorouter, by using `http://.` url as a service address. This gorouter accepts only http(s), in case you want to have tcp support you can use tcp-router [1].

      You can also build common virtual network for your apps and services located on different garden hosts using a tool called netman [2], (but it does not look like worth to mess with in your case).

      Here [3] is an example of Dockerfile for an image that you can push to CF with Diego running on garden-runc.

      Other useful docs you can find in cloud foundry docs [4].

      Hope it will help you.

      [1] https://github.com/cloudfoundry-incubator/cf-tcp-router
      [2] https://github.com/cloudfoundry-incubator/netman-release
      [3] https://gist.github.com/allomov/c78cbd23cc043d3852e0975e859da61c
      [4] http://docs.pivotal.io/pivotalcf/1-9/adminguide/docker.html

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!