Cloud Foundry Monitoring with Admin UI: Technical Overview
In this blog post, you’ll find a technical overview of Admin UI, a Web service that allows to gather metrics from the Cloud Foundry components. I’ve shed some light on its core layers, featured functionality, integration with CF APIs, etc. You’ll also discover what issues you may face while logging in, scaling, and debugging—as well as how to address them.
Technology basics (Ruby, the Cuba framework, Redis)
The metrics of Admin UI are based on data taken from the UAA, Cloud Controller databases, and NATS. Admin UI checks the /varz and /healthz endpoints of the Cloud Foundry components. Find more information about these endpoints in the following documentation:
You can read about the Admin UI dependencies in the Gemfile.
Admin UI is a Sinatra-based application that uses SQLite3 to cache data and store stats. To support multiple possible UAA and CC databases (currently, Cloud Foundry supports PostgreSQL and MySQL), Admin UI uses the Sequel gem. That’s why it requires some additional packets to be installed, including the PostgreSQL and MySQL client libraries.
With Admin UI BOSH release, you can expose Admin UI through the CF Router, using cf-registrar. Still, it is not the best way to do it.
Components and layers
UAA is used as an authentication tool in the latest version of Admin UI. When a user opens the application for the first time [link], he is redirected to the UAA endpoint with a login form. Then, a user is redirected back to Admin UI with username and access scope data in parameters. Username and scope are stored to a session and are further used to identify if a user has access to specific tabs.
The authentication process requires a special registration of Admin UI within UAA. In order to add Admin UI as the UAA client, you can use a script from Admin UI that provides access to UAA via the cf-uaac gem. If you install Admin UI with latest BOSH v4 release, you can run the errand jobs that will do it for you.
The Admin UI server has a special thread that updates the Cloud Foundry components data in the background. It has a low priority and doesn’t hamper the server’s work. This background thread gathers data from the Cloud Foundry components and stores it into SQLite3. Time period required for the thread to gather data is set in configurations. After data is saved to SQLite3, it is accessible by the main thread and is used in responses of a client. The server allows for performing some management tasks for the Cloud Foundry deployment, too. Since the Admin UI client access scope is used to get data or perform any management task, you will not be able to perform actions allowed to your user, but restricted for Admin UI.
Admin UI provides access to the CF components data and helps to manage Cloud Foundry deployment. Deeply integrated with Cloud Foundry, Admin UI brings forth loads of relevant information: a list of DEA components, a list of each DEA app containers, the UAA organizations, users and groups, statistics of resource consumption, etc.
Integration with Cloud Foundry (APIs)
To enjoy a full scope of possibilities offered by Admin UI, it should be connected to the following components:
NATS: the NATS client gem is used
the Cloud Controller REST API: no special library, all work with an API is implemented within Admin UI
the Cloud Controller DB URI: the sequel gem is used to connect the PostgreSQL or MySQL database
the UAA REST API: no special library, all work with an API is implemented within Admin UI
the UAA DB URI: the sequel gem is used to connect the PostgreSQL or MySQL database.
Performance / scaling
Admin UI is designed to work within the Ruby process, which implies strict constraints on scaling. While you can scale vertically without any limitations, horizontal scaling calls for using a load balancer with a sticky session. As a rule, Admin UI is used by a limited number of operators, so there is no data about request workload it can handle.
Error handling / logging and debugging
If you get “This page requires data from services that are currently unavailable,” it means that Admin UI doesn’t have access to the UAA or CC databases or is not able to get data from NATS.
To check if databases are available, you should SSH to the instance, where Admin UI is installed, and try to get access to the databases with client tools. Make sure Admin UI has correct settings for the databases, too. Learn more about this bug in this Github issue.
To find out if NATS is available, you need to get the NATS client and connect to NATS from within the Admin UI virtual machine. To do it, check out the NATS version in the Admin UI Gemfile and install a correspondent version of a gem. Then, you need to connect to NATS to prove it is possible:
gem install nats -v <nats-version> nats-sub ‘>’ nats://nats-user@nats-password@nats-host:nats-port
Another issue occurs, when you have multiple instances of Admin UI and the Load Balancer that works in the “round robin” mode (you can find details here). In this case, only the Admin UI header will be shown.
If you get the “Wrong Scope” message after authentication to Admin UI, it means you haven’t configured the Admin UI UAA client properly.
Localization (Chinese and Russian)
The Russian and Chinese languages support is implemented in the Altoros’s fork of Admin UI in the i18n branch. Pull request, with the changes to the official repository mentioned, is on approval to be merged.
You can deploy a version with Сhinese and Russian support, using our special BOSH release. Discover more about the Admin UI BOSH release with i18n in this blog post.
Issues / bugs
While using Admin UI, you may face the following issues:
The Admin UI server uses the SQLite3 file system database for caching information about Cloud Foundry. It may cause problems during horizontal scaling. See part “Performance / Scaling” to learn how to solve this issue.
The authentication process doesn’t recognize the changes made to the user access scope. It means if a user is logged in Admin UI, the result is the same, even after s/he is removed from DEA or his/her access scope is changed.
Admin UI proved to be a great tool with a strong CF bond, using not only the Cloud Foundry REST API, but databases of such core components as the Cloud Controller and UAA. Therefore, Admin UI is able to monitor the Cloud Foundry vital data in real-time without network overloading. Hope, this blog post casts light on what Admin UI is. Feel free to ask any questions in the comments.
Further reading: Creating a BOSH Release for Admin UI, a Monitoring Tool for CF7 Comments