Blog

How to Discover Services on Lattice with Consul

Alexey Zalesov

consul-latticeLattice is a light-weight, open source tool for clustering containers. Containers in a Lattice cluster are long-running processes or one-time tasks that are scaled and scheduled dynamically. Apps running in containers have to use external services, such as MySQL, RabbitMQ, etc., but if these services are dynamic, you cannot hardcode their IPs to the client.

The solution is to use a service discovery product, such as Consul, a highly available, distributed tool for discovering and configuring services. In this tutorial, I describe how an app running in Lattice can discover a MySQL service with Consul.


 

Prerequisites

First, you need to download three Vagrant boxes, which are approximately 2 GB in size. So make sure you have a good Internet connection. You will also need 1 GB of RAM for each machine (4 GB in total).

Here is what my environment looks like:

  • MacOS X 10.10.1
  • Vagrant 1.7.2
  • VirtualBox 4.3.28

You can try repeating the steps described in this tutorial on any configuration, but this one has been tested and it definitely works.
 

Creating a cluster

To get started, let’s spin up four virtual machines. We use Vagrant and VirtualBox to do this on a laptop (without any external infrastructure provider, like AWS). This makes deployment faster and much simpler.

$ vagrant up
  Bringing machine 'mysql01' up with 'virtualbox' provider...
  Bringing machine 'mysql02' up with 'virtualbox' provider...
  Bringing machine 'consul' up with 'virtualbox' provider...
  Bringing machine 'lattice' up with 'virtualbox' provider...

We need to open SSH sessions to MySQL and Consul VMs. You will be interacting with Lattice using the Lattice CLI, so you don’t need to SSH to it.

$ vagrant ssh mysql01
$ vagrant ssh mysql02
$ vagrant ssh consul
$ ltc target 10.0.0.13.xip.io
    Api Location Set

Now start your Consul cluster.

consul$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul \
-bind=10.0.0.10 -ui-dir /opt/ui -client=10.0.0.10

mysql01$ consul agent -data-dir /tmp/consul -bind=10.0.0.14 \
-config-dir /etc/consul.d -join 10.0.0.10

mysql02$ consul agent -data-dir /tmp/consul -bind=10.0.0.11 \
-config-dir /etc/consul.d -join 10.0.0.10

So, we have launched Consul in server mode on one node and in client mode on two more nodes. Now we need to verify that the cluster is running and the clients are registered in the Consul Web UI.

service-discovery-with-consul-web-ui

 

Registering the MySQL service

Once our Consul infrastructure is up and running, it’s time to register the MySQL service. This has already been done for you, so let’s just verify it.

To display the status of the MySQL service and make sure it is running, use:

vagrant@mysql01:~$ sudo service mysql status
    mysql start/running, process 1106

Then, connect to port 3306 using the CLI and check if it is listening to the port:

vagrant@mysql01:~$ mysql -h10.0.0.11 -P3306 -ulexsys --password=altoros
    mysql>

Finally, run a query against MySQL. We will use the following query to check database health:

mysql> select @@hostname;
    +------------+
    | @@hostname |
    +------------+
    | mysql01    |
    +------------+
    1 row in set (0.00 sec)

Check the service definition for MySQL.

vagrant@mysql01:~$ cat /etc/consul.d/mysql.json
{
  "service": {
    "name": "mysql",
    "port": 3306,
    "check": {
      "script": "/etc/consul.d/scripts/mysql_health.sh",
      "interval": "10s"
    }
  }
}

View the status of the MySQL service in the Consul Web UI.

service-discovery-with-consul-on-lattice-mysql

 

Discovering a MySQL service from Lattice

Let’s make sure that we can discover the service using DNS and HTTP interfaces.

vagrant@mysql02:~$ dig @127.0.0.1 -p 8600 mysql.service.consul SRV
  ;; ANSWER SECTION:
  mysql.service.consul.	0	IN	SRV	1 1 3306 mysql01.node.dc1.consul.
  mysql.service.consul.	0	IN	SRV	1 1 3306 mysql02.node.dc1.consul.

  ;; ADDITIONAL SECTION:
  mysql01.node.dc1.consul. 0	IN	A	10.0.0.14
  mysql02.node.dc1.consul. 0	IN	A	10.0.0.11


$ curl http://localhost:8500/v1/catalog/service/mysql | jq .
[
  {
    "ServicePort": 3306,
    "ServiceAddress": "",
    "ServiceTags": null,
    "ServiceName": "mysql",
    "ServiceID": "mysql",
    "Address": "10.0.0.11",
    "Node": "mysql02"
  },
  {
    "ServicePort": 3306,
    "ServiceAddress": "",
    "ServiceTags": null,
    "ServiceName": "mysql",
    "ServiceID": "mysql",
    "Address": "10.0.0.14",
    "Node": "mysql01"
  }
]  

We see that MySQL is running on two nodes whose IPs are 10.0.0.11 and 10.0.0.14.

Now, let’s push the Docker container with the application to Lattice. The container image is located on Docker Hub and is publicly available. There are a Consul agent alongside with a Go application inside. The application asks the local Consul agent what the MySQL service IPs are and chooses one of them randomly.

$ ltc create consul lexsys/consul-app -r -- "/opt/app.sh"
  ...
  Creating App: consul
  ...
  http://consul.10.0.0.13.xip.io

The next step is to point your browser to the application’s endpoint (`/demo`) and refresh several times. You will see that the application balances between the MySQL instances.

service-discovery-with-consul

service-discovery-with-consul-on-lattice

Let’s emulate failure of one service node. To do so, we will shut down one MySQL instance.

vagrant@mysql01:~$ sudo service mysql stop
    mysql stop/waiting

Our app doesn’t try to connect to the failed instance.

service-discovery-with-consul

We can see a failed health check in Consul’s Web UI.

service-discovery-with-consul-on-lattice-health-check

Now, start the MySQL service:

vagrant@mysql01:~$ sudo service mysql start
    mysql start/running, process 2576

This is it. Our application automatically learns that a new service instance is available and starts using it.

service-discovery-with-consul

service-discovery-with-consul-on-lattice

The code you will need to replicate this process can be found in this repository.
 


About the author: Aleksey Zalesov is a Cloud Foundry/DevOps Engineer at Altoros. He is an IBM Tivoli Certified Specialist with seven years of experience in system administration. Aleksey specializes in systems for IT monitoring and management, such as BOSH, a tool that can handle the full life cycle of large distributed systems, e.g., Cloud Foundry. His other professional interests include performance tuning techniques for Linux containers, etc.


Related video: Service Discovery on Lattice with Consul (CF Summit 2015)


Subscribe to our blog to get all the latest news on Cloud Foundry or follow @altoros.

Get new posts right in your inbox!

2 Comments
  • sowkarthika devan

    Nice article. How was the docker image created which has application alongside. Couldn’t find this in the repo link you posted.

    $ ltc create consul lexsys/consul-app -r — “/opt/app.sh”

    • lexsys

      Thank you! It looks like the Docker file is really missing in the repo, but since the post is 2 years old, I can’t find it, too. I would suggest starting from creating an Ubuntu image and then adding a compiled demo app on top of it—the app has no dependencies.

      Also, please note that Lattice is currently not actively maintained and may have become obsolete due to ongoing Cloud Foundry development.

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!