Big Data is Turning a Corner - and the road ahead is looking great

Having attended several big data conferences in the past few months, I have begun to notice something very profound. There are significantly fewer (almost zero) talks that focus simply on “What is Big Data?” or on “The 3 V’s of Big Data.”  There are many more presentations that dive into successful implementations, domain-specific use cases, useful applications, lessons learned, and tangible results.

Those of us who are so very immersed in the big data culture might have missed this important transition, which at first seemed subtle, but which now seems almost tangible.

The first nudge that I received about this shift came at the Innovation Enterprise’s Big Data Innovation Summit in Santa Clara, in April 2014. Talk after talk delivered something uniquely useful and informative.  I scribbled 16 pages of detailed notes in my conference notebook – and yet I barely scratched the surface of the deep valuable content that was delivered in those two days.  In particularly, the MapR presentation at the summit (by Anoop Dawar, MapR Senior Director of Product Management) inspired me to write the article H is for Hadoop, along with a Huge Heap of Helpful Big Data Capabilities.”

The second (and stronger) nudge that I received about the shift in big data status came at the Leverage Big Data Summit in San Diego, in May 2014. At the end of this intense (but intensely rewarding) 2-day event, during the final panel discussion of the conference, the audience members were asked to vote on where they thought we were on the Gartner hype cycle curve for big data.  Only about 32 attendees were still in the room for this non-scientific poll, which asked us to identify where we thought we were in the hype cycle:

1.     Technology Trigger – 0 votes

2.     Peak of Inflated Expectations – 9 votes

3.     Halfway between the Peak of Inflated Expectations and the Trough of Disillusionment – 15 votes

4.     Trough of Disillusionment – 0 votes

5.     Slope of Enlightenment – 5 votes

6.     Plateau of Productivity – 3 votes

Therefore, the majority of voters believed that we were neither at the peak of the hype cycle nor in the depths of despair and doubt about our data-driven predilections.  Though about half thought that we hadn’t reached bottom yet, nevertheless we had moved on, as a community and as a discipline. This was really good news.  The optimism was palpable. The self-justification and hand-wringing sessions were given no place on the conference agenda, and we were all quite happy to get on with the business of big data analytics – or, in that conference’s case, to get on with “leveraging big data”!  Following the conference, I was inspired to write the article Taking Big Data to the Next Level,” which reviewed the many significant lessons learned and words of wisdom that Colin Dover (of SAP) presented at the conference.  I think that the #1 take-away from Colin’s talk is this quote: “Culture eats strategy for lunch.” The shift to a positive forward-looking big data world view now suggests that the culture issue (though still quite important and significant – i.e., that most corporate cultures still have not embraced big data analytics as a way of life) will be less negatively critical and more constructively critical. (Oh yes, how many pages of notes did I scribble in my notebook at this 2-day conference? 26.)

The third and most convincing insight that I received to convince me that this shift to a more productive (and less hyped, less doubting) approach to big data analytics had indeed occurred was at the Useful Business Analytics Summit in Boston, in June 2014. The title of the conference said it all – the focus was on useful big data analytics (i.e., the uses, use cases, user experiences, usage results, useful applications).  The conference speakers came from many different verticals:  travel, entertainment, sports, broadcasting, finance, education, retail, insurance, automotive, social media, manufacturing, telecomm, energy, Internet businesses, and more.  The business case and the business value of big data analytics were apparent and transparent in each presentation.  Several very interesting articles were written about the conference topics leading up to the event and after the event – these articles provide additional insights, advice, lessons learned, recommendations, tips, and interviews (including one with your humble correspondent).

I was able to attend only one day of the Useful Business Analytics Summit, and yet I captured 16 pages of notes in my conference notebook.  For those of you keeping score, my note-taking rate from the three conferences mentioned above increased monotonically with time:  8 pages per day for the first conference; 13 pages per day for the second; and 16 pages per day for the third.  That trend further reveals the increasing value and usefulness of the stories being told and the solutions being developed by today’s big data analytics practitioners.  For a thorough review of the big data Hadoop solution offered by MapR, see the informative March newsletter and check out the awesome new MapR Apps Gallery.  With such useful tools as these, applied to your own use cases, you too can turn the corner with Big Data and achieve its promised potential on the plateau of productivity.

Streaming Data Architecture:

New Designs Using Apache Kafka and MapR Streams

 

 

 

Download for free