Exciting developments in Spring for Apache Hadoop project

-->

Few days back, Spring source community announced the second milestone of SHDP (Spring for Apache Hadoop) Project. The very first time I had come to know about this initiative from VMWare, I was almost sure that this is going to be a boon for Java spring development community interested in Big data. In the second milestone, they have made it a point to address the major challenges of developers by providing many of the Spring’s powerful concepts for Apache Hadoop development.

Gradually this project is evolving and rightly hitting the main challenges of developers who are working on the Hadoop and its peripheral technologies. The new milestone not only covers the core support for Hadoop based MR Jobs but also provides support for related technologies like HBase, Hive, Pig etc.

For the developers who are using HBase in their stack, this release is a wow moment. Spring has offered its powerful DAO (Data Access Object) concept for HBase. Developers can now use the powerful template for HBase crud without worrying about exception handling, writing boilerplate code or, handling resources and disposing them. This means that you don’t need to worry about tedious tasks like looking up a HBase table, running the query, building a scanner, cleaning up resources etc...

In case you really want to get a feel of how much headache it removes for a developer, here is a sample code snippet provided by Spring source to read each row from a HBase table.

// read each row from 'MyTable' List rows = template.find("MyTable", "SomeColumn", new RowMapper() {   @Override   public String mapRow(Result result, int rowNum) throws Exception {     return result.toString();   } }));

This definitely reduces a lot many lines of code. The other noticeable point is that all different types of exceptions being raised by the underlying HBase APIs are being converted into one DataAccessException which again eases the development for layers involved in CRUD operations with HBase. So the developers who are using HBase in their technology stack are destined to enjoy this release.

There is a lot of good news for the development community who is using Hive and Pig. Spring seems to have support of both clients (Thrift and JDBC) for Hive. For the Thrift client a dedicated namespace has been provided. In case you want to use the JDBC client, one can leverage the rich JDBC support in Spring which provides you facilities like Jdbc template, NamedParameterJdbcTemplate etc. For the pig developers, Spring provides easy configuration and instatiation of pig servers for registering and executing pig scripts either locally and remotely.

There is a lot more like cascading integration, security support for which I am excited to delve into details, but one thing is quite clear that the small elephant is definitely going to enjoy this spring :)

Here are some of the useful links in case you are interested to get into the details of all these developments:

Blog post on 2nd milestone release of Spring for Apache Hadoop

Spring Data Project

Spring Data on Apache Hadoop

MapReduce v/s SQL

-->

I was exposed to the MapReduce paradigm couple of years ago and am in touch with the open source implementation of MapReduce framework (Hadoop) since then. We started to play with Hadoop actively to understand the pros and cons of the framework and as of today we have considerably progressed by building our new platform backed by this powerful distributed computing framework.

During this whole journey, we encountered many challenges and questions. One of the most frequent query was why not SQL based systems and why Hadoop? After going through various discussions, technical articles and my own exposure to this system, I thought of sharing my own experience around this very frequent query originating from different development community.

The concept of MapReduce was actually brought in to solve the scale problem when you are dealing with huge amount of data, which nowadays is not a problem that exists only for big corporate houses (like Google, Microsoft etc.). With the bloat of information everywhere, this problem of dealing with huge data set has become a more common problem than earlier.  Most of the times people try to solve it with the conventional systems like SQL. Many of the places, it has been successful up to some extent, but beyond a threshold, it becomes really very challenging to tackle this problem with those conventional systems. There are many reasons of this challenge.

1. Unstructured data processing: SQL systems are optimized to deal with structured data. The whole concept of relational databases is based on the notion of relational schema to store information, which becomes a challenging when you are dealing with unstructured data. In such cases you end up retro fitting your information in a tight schema, which creates challenge while digging out insight from the data. 

2. Control on processing steps: SQL mostly is a language, which is declarative in nature, and when you use SQL for querying information, you pass on your interest in result and the data sources from where this information can be retrieved. The actual details of how to get to the result still lies under the control of query processing engine. So, you are left with nothing (Most of the time you can only pass on some clue/hint to the engine to influence its strategy of processing), but to rely upon the genius of processing engine to provide you the data through optimal processing. 

3. Scale (out v/s up): Conventional systems like relational databases are designed to work on monolithic systems rather than distributed clusters. This imposes the constraint that while addressing the scale issue, you will end up buying costlier hardware. One thing worth noting here is that cost of the hardware does not increase linearly, meaning the cost of one machine having 5 times power than a standard machine is more than 5 times costlier than the standard machine. Because of this equation, it makes more sense to address the scale challenge horizontally than vertically. That means if you build framework where in you can scale by just adding more no.  of machines, it makes more sense. Of course adding more machines comes with an overhead of more co-ordination requirements. For which if you have at your luxury a framework like Hadoop, then it can be a smart move to approach for scale out using such frameworks.

4. Offline v/s online processing: The original requirement because of which the MapReduce framework originated was around processing huge amount of data without caring about items like realtime processing, transaction support etc. Hence these systems are more optimized for offline processing rather than realtime processing of data. Though there are many other technologies, which are trying to address these pieces as well, but still as of today, in the heart of the framework, it is fundamentally a offline distributed processing framework. 

5. Raw talent v/s conventional wisdom:  Relational database systems have existed in the industry for quite a few years and it has catered to the need of its age very successfully. These successful years of SQL adoption in the s/w industry has produced a lot of experts around that technology.  So, when you are using SQL based systems, you have the luxury to have these expert advices. On the contrary, in the MapReduce world, the thought process is quite different and during your initial days of its adoption, it is very likely that you may end up designing your system in the relational way even though underlying you tend to use these new frameworks. Design and thought process in MapReduce requires a raw thought process, and the moment you try to retrofit this with conventional thought process of how to build your stack thinking about entities and its relationships, you debar yourself from gaining the juice of MapReduce framework.  

Conclusion

This post certainly does not cover each and every aspect of both the systems, but IMHO, it provides you some data point to think about while you are planning to build your stack around these technologies. Obviously, the intent of the post is not to prove anything, but to dig out different relevant points around these technologies. For a given requirement, it is even possible (and it is not very remote possibility in many of the cases) that you end up building your stack with a marriage of both these technologies.

Signing off for right now with a request to provide your fruitful comments…

--RBK