How to Discover Services on Lattice with Consul

by Aleksey ZalesovJune 26, 2015
Learn how to create a cluster, register the MySQL service, and discover it from Lattice.

consul-lattice

Lattice is a light-weight, open-source tool for clustering containers. In a Lattice cluster, containers represent long-running processes or one-time tasks that are scaled and scheduled dynamically. Apps running in containers have to use external services, such as MySQL, RabbitMQ, etc., but if these services are dynamic, you cannot hardcode their IPs to the client.

The solution is to use a service discovery product, such as Consul, a highly available, distributed tool for discovering and configuring services. In this tutorial, we describe how an app running in Lattice can discover a MySQL service with Consul.

 

Prerequisites

First, you need to download three Vagrant boxes, which are approximately 2 GB in size. So, make sure you have a good Internet connection. You will also need 1 GB of RAM for each machine (4 GB in total).

In our case, the environment is:

  • MacOS X 10.10.1
  • Vagrant 1.7.2
  • VirtualBox 4.3.28

You can try repeating the steps described in this tutorial on any configuration, but this one has been tested, and it definitely works.

 

Creating a cluster

To get started, let’s spin up four virtual machines. We use Vagrant and VirtualBox to do this on a laptop (without any external infrastructure provider, like AWS). This makes deployment faster and much simpler.

$ vagrant up
  Bringing machine 'mysql01' up with 'virtualbox' provider...
  Bringing machine 'mysql02' up with 'virtualbox' provider...
  Bringing machine 'consul' up with 'virtualbox' provider...
  Bringing machine 'lattice' up with 'virtualbox' provider...

We need to open SSH sessions to MySQL and Consul VMs. You will be interacting with Lattice using the Lattice CLI, so you don’t need to SSH to it.

$ vagrant ssh mysql01
$ vagrant ssh mysql02
$ vagrant ssh consul
$ ltc target 10.0.0.13.xip.io
    Api Location Set

Now, start your Consul cluster.

consul$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul \
-bind=10.0.0.10 -ui-dir /opt/ui -client=10.0.0.10

mysql01$ consul agent -data-dir /tmp/consul -bind=10.0.0.14 \
-config-dir /etc/consul.d -join 10.0.0.10

mysql02$ consul agent -data-dir /tmp/consul -bind=10.0.0.11 \
-config-dir /etc/consul.d -join 10.0.0.10

So, we have launched Consul in a server mode on one node and in a client mode on two more nodes. Now, we need to verify that the cluster is running, and the clients are registered in the Consul Web UI.

service-discovery-with-consul-web-ui

 

Registering the MySQL service

Once our Consul infrastructure is up and running, it’s time to register the MySQL service. This has already been done for you, so let’s just verify it. To display the status of the MySQL service and make sure it is running, use the command below.

vagrant@mysql01:~$ sudo service mysql status
    mysql start/running, process 1106

Then, connect to port 3306 using the CLI and check if it is listening to the port.

vagrant@mysql01:~$ mysql -h10.0.0.11 -P3306 -ulexsys --password=altoros
    mysql>

Finally, run a query against MySQL. We will use the following query to check database health.

mysql> select @@hostname;
    +------------+
    | @@hostname |
    +------------+
    | mysql01    |
    +------------+
    1 row in set (0.00 sec)

Check the service definition for MySQL.

vagrant@mysql01:~$ cat /etc/consul.d/mysql.json
{
  "service": {
    "name": "mysql",
    "port": 3306,
    "check": {
      "script": "/etc/consul.d/scripts/mysql_health.sh",
      "interval": "10s"
    }
  }
}

View the status of the MySQL service in the Consul Web UI.

service-discovery-with-consul-on-lattice-mysql

 

Discovering a MySQL service from Lattice

Let’s make sure that we can discover the service using DNS and HTTP interfaces.

vagrant@mysql02:~$ dig @127.0.0.1 -p 8600 mysql.service.consul SRV
  ;; ANSWER SECTION:
  mysql.service.consul.	0	IN	SRV	1 1 3306 mysql01.node.dc1.consul.
  mysql.service.consul.	0	IN	SRV	1 1 3306 mysql02.node.dc1.consul.

  ;; ADDITIONAL SECTION:
  mysql01.node.dc1.consul. 0	IN	A	10.0.0.14
  mysql02.node.dc1.consul. 0	IN	A	10.0.0.11


$ curl http://localhost:8500/v1/catalog/service/mysql | jq .
[
  {
    "ServicePort": 3306,
    "ServiceAddress": "",
    "ServiceTags": null,
    "ServiceName": "mysql",
    "ServiceID": "mysql",
    "Address": "10.0.0.11",
    "Node": "mysql02"
  },
  {
    "ServicePort": 3306,
    "ServiceAddress": "",
    "ServiceTags": null,
    "ServiceName": "mysql",
    "ServiceID": "mysql",
    "Address": "10.0.0.14",
    "Node": "mysql01"
  }
]  

We see that MySQL is running on two nodes, which IPs are 10.0.0.11 and 10.0.0.14.

Now, let’s push the Docker container with the application to Lattice. The container image is located on Docker Hub and is publicly available. There are a Consul agent alongside with a Go application inside. The application asks the local Consul agent what the MySQL service IPs are and chooses one of them randomly.

$ ltc create consul lexsys/consul-app -r -- "/opt/app.sh"
  ...
  Creating App: consul
  ...
  http://consul.10.0.0.13.xip.io

The next step is to point your browser to the application’s endpoint (`/demo`) and refresh several times. You will see that the application balances between the MySQL instances.

Below is the output for port 10.0.0.11.

service-discovery-with-consul

And the similar one for port 10.0.0.14.

service-discovery-with-consul-on-lattice

Let’s emulate failure of one service node. To do so, we will shut down one of the MySQL instances.

vagrant@mysql01:~$ sudo service mysql stop
    mysql stop/waiting

Our app doesn’t try to connect to the failed instance.

service-discovery-with-consul

We can see a failed health check in the Consul’s Web UI.

service-discovery-with-consul-on-lattice-health-check

Now, start the MySQL service.

vagrant@mysql01:~$ sudo service mysql start
    mysql start/running, process 2576

This is it. Our application automatically learns that a new service instance is available and starts using it. Below is the output for port 10.0.0.11.

service-discovery-with-consul

Likewise, you get the output for port 10.0.0.14.

service-discovery-with-consul-on-lattice

To replicate this process, use the source code from this GitHub repository.

 

Related videos

In this video, Juan Pablo Genovese and Aleksey Zalesov talk about enabling service discovery with Lattice during the Cloud Foundry Summit 2015.

In this video, Alexey explains how to discover MySQL on Lattice via a console means.

 

Further reading

 

About the author

Aleksey Zalesov is a Cloud Foundry/DevOps Engineer at Altoros. He is an IBM Tivoli Certified Specialist with seven years of experience in system administration. Aleksey specializes in systems for IT monitoring and management, such as BOSH, a tool that can handle the full life cycle of large distributed systems, e.g., Cloud Foundry. His other professional interests include performance tuning techniques for Linux containers, etc.


: Service Discovery on Lattice with Consul (CF Summit 2015)


This post was written by Aleksey Zalesov and edited by Sophia Turol and Alex Khizhniak.