Taking Big Data to the Next Level

I attended the Leverage Big Data 2014 Summit recently, sponsored by Tabor Communications (whose brands include Datanami, EnterpriseTech, and HPCWire) and run by nGage Events.  One of the keynote talks was presented by Colin Dover of SAP.  The subtitle of Colin’s Big Data presentation was “Sensors, Signals, Swans, and Shockwaves”.  He presented so many excellent ideas and insights about the current state of big data analytics adoption in business that I believe that it would be useful to review some of his talk’s highlights here.

First, let’s pick apart his title:

  • Sensors – as we described in a previous article, big data can be described as “everything, quantified and tracked.”  This reflects the ubiquitous presence of sensors everywhere.  These may include surveillance cameras in many places (airports, banks, shopping malls, major cities’ streets), but it also refers to our own mobile devices (phones and pads) that serve as a sensor on our lives (emitting data about what we do, where we visit, what we buy, whom we talk to, what we say, etc.), or web usage logs, or social media comments, or customer reviews (online, in email, or via the call center).  Sensors are increasingly watching and tracking everything. Even in science this is true – sensors monitor our health, our workouts, the oceans, the weather, Earth’s seismic activity, the Universe, and more.  The deployment of sensors, use of sensors, data acquisition from sensors, and/or mining of data from multiple sensors represent a very big challenge to businesses who are looking to derive insights and value from these diverse data streams.
  • Signals – one of the biggest challenges of big data (and one of the bottlenecks for business analytics projects) is extracting business-relevant and business-valued signals from the noise.  This can be achieved without too much difficulty at first, by establishing certain rules or filters that are applied to the data streams – e.g., to detect certain types of events, or changes in behavior, or emergence of new patterns, or an increase/decrease in the frequency of specific types of patterns (including web usage patterns, or customer buying patterns, or manufacturing defects). More sophisticated algorithms (business rules) can be applied as it becomes clear that they bring value, deliver ROI, and satisfy your business objectives. An example of a more sophisticated approach would be to deploy predictive analytics algorithms autonomously on the incoming data stream (such as Markov models, or regression models, or neural networks) and/or to integrate multiple diverse data types in the algorithm (e.g., using sentiment analysis on social media comments, while also measuring visits to your website, responses to marketing campaigns [coupons redeemed], and sales numbers).
  • Swans – this refers to “The Black Swan”, which was the subject of a book with the same title by Nassim Taleb.  Black swans are rare (or improbable) events that often have great consequences.  These are outliers, which are objects/events that are outside the bounds of your expectations, and thus unpredictable. In risk management, one attempts to identify such black swans or at least to plan for them. The value of big data is that it may actually contain many examples of such improbable events – the rare one-in-a-million or one-in-a-billion occurrence, which is now visible specifically because you have collected billions of measurements (signals from your sensors).
  • Shockwaves –we can imagine all sorts of shockwaves in business. In the context of big data, these may be the technological consequences of ingesting massive data into your systems, or the operational changes in your business processes that the new analytics demands, or the cultural shockwaves in corporate boardrooms with the arrival of the Chief Data Scientist and/or Chief Data Officer, or the revolutionary changes in the skills that you need to hire in order for your business to survive in the brave new world of big data analytics.  Colin said it this way: “global economic fragility is the new normal.”

After this overview, there were several significant lessons learned and words of wisdom that Colin Dover presented in his talk for businesses seeking success in big data analytics, including these nuggets:

  • (Anonymous quote): “The Stone Age was marked by man’s clever use of crude tools; the Information Age, to date, has been marked by man’s crude use of clever tools.”
  • “Beware the HiPPO” (Highest Paid Person‘s Opinion) in the corporate boardroom when planning your big data strategy.  Your business strategy and actions should be data-driven, not based on past experience or gut instinct. Big data analytics is a whole new ballgame.
  • “In 64% of enterprises, fewer than 10% of decision-makers use analytics.”
  • What constitutes Data-Driven Decision-Making?
  1. Data analytics techniques (data science) 
  2. Big data technologies (e.g., Hadoop, NoSQL, graph databases)
  3. Visualization methods
  • The two biggest changes in the big data era are:
  1. User transformation
  2. Cultural transformation
  • “Culture eats strategy for lunch” – i.e., your business big data strategy is likely to fail because the naysayers (the guardians of your corporate culture) won’t let a new thing succeed or upset the status quo.
  • Old-school Business Intelligence reporting (descriptive analytics for hindsight, “what did happen?”) just won’t cut it anymore.  Analytics is now going far beyond hindsight:
  1. What is happening (agile visual analytics; oversight)?
  2. What will happen (predictive analytics; foresight)?
  3. What is the best that could happen (prescriptive analytics; optimization; insight)?
  4. Can the big data analytics system learn to adapt to new situations, new requirements, new sensors, new signals, black swans, and shockwaves autonomously (cognitive analytics; enlightenment)?
  • Dark data may be your biggest form of big data – dark data are those files, databases, reports, customer logs, transaction logs, etc. that fill your data storage network, but have heterogeneous formats, poor metadata, uncertain provenance, questionable data quality, weak governance, unknown value, and untapped potential.
  • Consequently, every company needs a data scientist, in order to:
  1. Unlock the business’ dark data
  2. Unlock the value in the business’ dark data
  • Business’ biggest big data challenges can be summarized as:
  1. Data governance
  2. Staffing and skills for big data analytics operations
  3. Cost
  4. Uncertain value of big data
  5. Unclear technical requirements on analytics
  6. Connecting the right people to the right information
  7. Turning signals (from sensors) into business value
  8. Preparing for the black swans and shockwaves in a cost-constrained world
  9. Getting ready for the next big wave: Smart Everything! (including the Internet of Everything, and Machine-to-Machine intelligence)
  10. Having a mindset that focuses on “The Art of the Possible” (e.g., the revolutionary evolution from the tethered rotary phone, to the first clunky untethered mobile phone, to the omnipresent ultra-high-capability smart phone).

In the end, Colin Dover offered these 5 pieces of advice in his Leverage Big Data Summit presentation to businesses who are struggling with their big data analytics programs:

  1. Shift now to a big data mindset;
  2. Develop your business’ strategic big data analytics use case(s) that will differentiate your business from others;
  3. Prepare yourself for continuous disruption (beware the HiPPO in the room!);
  4. Focus on analytics value, not on big data volume; and
  5. Create and nurture an information-centric business culture at all levels, everywhere in your company.

Analytics is not just for power users – it is for everyone!  As someone said a few years ago, you should believe in, live by, and act on this mantra: Data Science for the Masses!”

Streaming Data Architecture:

New Designs Using Apache Kafka and MapR Streams




Download for free