Building a MapReduce PoC
Most customers prefer to validate a concept before investing into a production system. In our case, an ultimate solution should allow for long-running computations, have a web interface for setting tasks, and support MapReduce or other data processing methods. So, an ideal prototype would be a platform that provides a simple method for deploying, scaling, and monitoring apps. That is what Iron Foundry does.
In this post, I’ll describe how to create a prototype with Iron Foundry on Cloud Foundry. My test application will use MapReduce to find the most popular words in a text. The picture below demonstrates the data processing workflow.
As you can see, text is transferred from a client to a web component for processing. At this stage, the job is divided between available Mappers. Mappers send the results to a Reducer, which performs final computations and returns the output back to the client via a Notifier. Below, I will describe how to create each of the components, establish communication between them, and deploy an application with Iron Foundry.