Blog

.NET on Cloud Foundry, Part 2: Prototyping with Iron Foundry

Alex Makarenko

Part 1 | Part 2 | Part 3

Building a MapReduce PoC

Most customers prefer to validate a concept before investing into a production system. In our case, an ultimate solution should allow for long-running computations, have a web interface for setting tasks, and support MapReduce or other data processing methods. So, an ideal prototype would be a platform that provides a simple method for deploying, scaling, and monitoring apps. That is what Iron Foundry does.

In this post, I’ll describe how to create a prototype with Iron Foundry on Cloud Foundry. My test application will use MapReduce to find the most popular words in a text. The picture below demonstrates the data processing workflow.

As you can see, text is transferred from a client to a web component for processing. At this stage, the job is divided between available Mappers. Mappers send the results to a Reducer, which performs final computations and returns the output back to the client via a Notifier. Below, I will describe how to create each of the components, establish communication between them, and deploy an application with Iron Foundry.

 

Hosting web apps on Cloud Foundry with Iron Foundry

So, let’s start with the first component of the system—a website. It is easy to push a standard MVC application to Iron Foundry using IIS Hostable Web Core. I didn’t even have to change the code or do additional configuration. But what about hosting web applications with Iron Foundry?

My web application will not depend on IIS and—most probably—it will use HTTP Listeners to handle traffic. Keep it in mind that you should know port numbers to enable listening to new HTTP requests. I learned how to do it in the Iron Foundry Google group:

 

var _listener = new HttpListener(); var port = Environment.GetEnvironmentVariable("VCAP_APP_PORT"); _listener.Prefixes.Add("http://*:" + port + "/");

 
So, Cloud Foundry router will redirect all requests to the VCAP_APP_PORT. It is also necessary to use a “http://*:port/” pattern to follow the Iron Foundry firewall rule.

 

Enabling messaging with MongoDB’s capped collections

Now we should provide communication between all components of a system. If we take Microsoft Azure, Message Bus is used as a messaging infrastructure. In the current version of ironfoundry.me, there are only 2 services: MS SQL and MongoDB and no messaging options are provided. Definitely, I can use RabbitMQ-as-a-Service but its free plan does not allow me to have more than 3 connections and it lacks some other features my demo needs. I knew that it is not a great deal to build a custom messaging component based on the MongoDB Capped Collections. To test this approach one should know how to:

  • add a MongoDB service
  • get a connection string

Adding MongoDB is simple, all you need is execute two commands in the command line interface: create a service and bind a service. Below, you will see an example of creating a my-mongo service and binding it to the Mapper app.

After the binding, the VCAP_SERVICES environment variable will contain all necessary details to establish a connection.

const string envName = "VCAP_SERVICES"; var settings = Environment.GetEnvironmentVariable(envName); var jmongo = JObject.Parse(settings); var db = jmongo["mongodb-2.2"][0]["credentials"]["db"]; var url = jmongo["mongodb-2.2"][0]["credentials"]["url"]; var client = new MongoClient(url.ToString()); var server = client.GetServer(); var database = server.GetDatabase(db.ToString());

 
Unfortunately, my first attempts to launch my own messaging service failed. I was able to connect, but when I was trying to start any activity, I got a message: “System.IO.EndOfStreamException: Attempted to read past the end of the stream.” As I have already tested this solution locally and it worked well, I was not really inspired by the idea to start creating a new one over again. Here I should do justice to ironfoundry.me developers. They are open for communication and listen to the community. So, very soon they adjusted a default configuration setting on the mongodb deployment, which resolved this issue.

However, there is another way of adding a messaging service. I found MongoDB-as-a-Service in the list of Cloud Foundry services. This solution has a free plan with lots of capabilities. So, I have just created a new database and successfully run my Iron Foundry MongoDB Queue test application.

Next time, I will tell how to create a notifier using WebSockets and show how to build a simple MapReduce app in Iron Foundry.

 

Implementing a notifier with WebSockets

In the beginning, I showed the architecture of the system. As you can see, a Notifier app pushes real-time updates to the client via WebSockets. There are many frameworks that provide the required functionality; I opted for XSockets.NET, a framework for building real-time .NET apps.

I needed to create a self-hosted XSockets server in Iron Foundry and establish communication between it and the HTML client. Unfortunately I failed to negotiate a handshake with XSockets in Iron Foundry, since Cloud Foundry router had some issues with the WebSockets. Then, I came up with another idea of starting a Node.js HTTP server and attaching a socket.io instance to it.

var host = process.env.VCAP_APP_HOST || "localhost"; var port = process.env.VCAP_APP_PORT || 3001;   var server = require('http').Server(); var io = require('socket.io'); server.listen(port, host); io.listen(server);


It worked out and I became able to access the HTML client. However, one more thing was left—the server should be accessible from the .NET Reducer console app. That is why I was to use socket.io v0.9 instead of sockets.io v1, which was not supported by the Socket.IO library. So, the notifier was implemented as a Node.js server in combination with the socket.io package v0.9.

 

Assembling a test app for Iron Foundry

At this stage, you should put all components into a single system. Our web client is a simple HTML page developed with Angular and Bootstrap. It looks like this:

2

We will host the page and services via ServiceStack. To start such a web component in Iron Foundry, you need to get a VCAP_APP_PORT variable:

var uri = "http://*:" + Environment.GetEnvironmentVariable("VCAP_APP_PORT") + "/"; var host = new AppHost();   host.Init(); host.Start(uri); "Type Ctrl+C to quit..".Print();   Thread.Sleep(Timeout.Infinite);

The Start button will send text to the server where it will be divided into several parts and propagated to the Mapper applications.

var parts = _mappers.Count; var chunks = Helper.SplitText(req.Text, parts); for (var i = 0; i < parts; i++)   SendChunk(_mappers[i], chunks[i]);

Where SendChunk:

var queue = new MongoQ(Constants.QueueLength, mapper); queue.Send(new Message(chunk));


 

Implementing a word search algorithm using MapReduce

The main goal of my algorithm was finding a word that has been used most frequently in a text. Planning logic of a prototype app in Iron Foundry is the same as on any other platform, so the steps described in this post are quite universal for all MapReduce workloads.

The diagram below illustrates how the MapReduce algorithm works.

Basically, the test task can be represented by two simple unit tests:

[TestMethod] public void MapReduce() { const string text = "a bb ccc ccc"; const string result = "a:1,bb:1,ccc:2,"; var map = Helper.MapReduce(text); Assert.AreEqual(result, map); } [TestMethod] public void Reduce() { const int max = 3; var words = new List> { new KeyValuePair("a", 1), new KeyValuePair("a", 2), new KeyValuePair("b", 2) }; var reduce = Helper.Reduce(words).ToList(); Assert.AreEqual(reduce[0].Value, max); }

You can use the following code to implement Mapper tasks in C#:

text.Split(new[] { " ", ".", ",", "\n", "\r\n", "*" }, StringSplitOptions.RemoveEmptyEntries) .Select(item => new KeyValuePair(item, 1)) .GroupBy(k => k.Key, v => v.Value).Select(s => new KeyValuePair(s.Key, s.Count())) .Aggregate("", (current, word) => current + (word.Key + ":" + word.Value + ","));

As an alternative, if you prefer functional style, you can create Mapper tasks with F#:

let words input= Regex.Matches(input, "\w+") |> Seq.cast |> Seq.map (fun (x:Match) -> x.Value.ToLower()) |> Seq.countBy id |> Seq.reduce (fun (a, b) (c, d) -> (a + ":" + b.ToString() + "," + c + ":" + d.ToString() + ",", 1)) |> fst

 
Or

let words (input:string) = input.Split([|" "; "."; ","; "\n"; "\r\n"; "*"|], StringSplitOptions.RemoveEmptyEntries) |> Seq.cast |> Seq.countBy id |> Seq.reduce (fun (a, b) (c, d) -> (a + ":" + b.ToString() + "," + c + ":" + d.ToString() + ",", 1)) |> fst

Since the C#-based solution worked a little bit faster under the workload used for a prototype app, I will opt for it. Now, I am not going to research into how to improve performance, since it is a topic for a separate blog post.

For reducing operation, we can use this approach:

allWords.GroupBy(k => k.Key).Select(s => new KeyValuePair(s.Key, s.Sum(p => p.Value))).OrderByDescending(o => o.Value);

 
So, the preparations are over and now we should think how an app will process Mapper and Reducer tasks. The first component will map the text and send it. You can implement Reducer using the reactor pattern. It has Event Demultiplexer, which will display names of Mappers ready for sequential processing.

var demultiplexer = new MongoQ(Constants.QueueSmall, Constants.Demultiplexer); while (true) { var availableMappers = demultiplexer.Receive(); ProcessMessage(availableMappers); }

 
In the next section, I will tell you how to assemble all parts of the prototype into a single solution. You will also learn how to deploy a MapReduce app to Iron Foundry.

 

Deploying applications to Cloud Foundry with Iron Foundry

In this section, I will show you how to deploy an application that consists of four major components: a Web site, a Mapper, a Reducer, and a Notifier. The picture below illustrates the workflow for processing a request from a client application. Each of the components can be scaled horizontally.

Our main goal is to deploy the system as easily and quickly as possible. You need to copy binary output with a post-build event for each application to a folder with the following structure:

-deploy -web -mapper -reducer -notifier

 
After that, add the manifest.yml file to the deploy folder and run the CF PUSH command. The manifest will provide all the information necessary for the cloud solution to deploy the system. The manifest.yml file looks like this:

--- applications: - name: ironicweb memory: 256MB stack: windows2012 instances: 2 path: ./web/ - name: mapper memory: 256MB stack: windows2012 instances: 4 no-route: true path: ./mapper/ - name: reducer memory: 256MB stack: windows2012 instances: 1 no-route: true path: ./reducer/ - name: ironicnotifier buildpack: https://github.com/cloudfoundry/heroku-buildpack-nodejs.git memory: 256MB instances: 1 path: ./notifier/ command: node notifier.js

 
Please pay attention to the “stack” element. It should be windows2012 for .NET applications. If your application is designed to run as a background worker, you must specify “no-route: true.” Once this setting is enabled, the Iron Foundry component will stop trying to connect to the application to check its health status.

Since Notifier is not a .NET application—it uses a slightly different manifest file—I have added a buildpack with Node.js. It specifies what runtime must be prepared for the application. In the manifest file, I added a command that will execute the Node.js file. Remember to add a package.json file to the Notifier folder. This file must contain details about the deployed application:

{ "name": "notifier", "version": "0.0.1", "dependencies": { "socket.io": "0.9.17" }, "engines": { "node": "0.10.x", "npm": "1.3.x" } }

 
After you have pushed your applications, open ironicweb.beta.ironfoundry.me to make sure that all of your application instances (10 in my system) work together and respond to requests.

 

Validating a prototype in Iron Foundry

In this section, I will share my experience with Iron Foundry’s integration capabilities. At this stage, we have already used Iron Foundry to deploy our product on Cloud Foundry. There are 4 connected micro services horizontally scaled to 8 instances.


 
Two of the components can be accessed via HTTP using these URLs: ironicweb.beta.ironfoundry.me and ironicnotifier.beta.ironfoundry.me (the second one is a node HTTP server for socket communication without a UI). IronicWeb helps to find the most popular words in texts. The Web interface allows for entering text manually or via a simple file upload.

The first step after the web page has loaded is to discover mappers. To do it, IronicWeb uses a custom MongoDB messaging component. This enables us to scale mappers via the command line interface and ensures that the new map-reduce job will use all available resources.

To re-deploy the entire system, we must run just one simple “cf p” command and wait for staging to finish, which takes less than one minute. There is no need to worry about the environment and various settings—we can just code and enjoy the result.

In the next post, I will explain how to move the whole system to the original Cloud Foundry platform without Iron Foundry. In addition, I will go over the current options and limitations for.NET products on CF.

 

Further reading


Get notified when new issues of "Iron Foundry for CF" series go live:


This series was written by Alex Makarenko; edited by Volha Kurylionak and Alex Khizhniak.

No Comments

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!