In this year’s Gartner hype cycle analysis, big data is approaching the trough of disillusionment. Does that mean it is unlikely for organizations to succeed with big data adoptions and should they focus elsewhere? No, of course not. Marketing has done its job and everybody has heard of big data, though few know what it means for their organization and how to approach the topic. Yet, the answer to how to succeed with big data is surprisingly straightforward once we ignore the hype and focus back on the basics.
A common big data challenge for new adopters is a confusion of technology and strategy. Organizations have educated themselves about big data or have organically grown proof of concepts in house. In both cases, the conversation quickly narrows down to the ubiquitous Hadoop and subsequently equivalating it with big data. Similarly, deploying Hadoop to hopefully achieve an aspect of a business goal becomes the replacement for a big data strategy. That surprisingly common story is, of course, the reverse of what the process should be, and usually the first step to reduce risk and focus on value is to turn this upside down.
A top down approach is common sense but the complexity and novelty of big data and the broad, fast evolving technology involved lead even seasoned managers astray. Hadoop alone has an overwhelmingly rich ecosystem of supporting applications as well as competing solutions from various vendors. Beyond Hadoop, the related business intelligence, visualisation, streaming, in-memory computation and the huge, diversified NoSQL space ensure that any non-expert is confused instantly.
The solution is to start with the business goals, which usually are clear, and then discover how big data may fit into the picture as a strategy. This regularly needs education, research and support from external experts who have experience and access to industry specific and cross-industry big data trends. Importantly, this step should be a collaboration to contextualise big data patterns and technologies with the domain knowledge and business goals. Ideally, no solution, technology or vendor should drive the conversation at this point.
The outcome of the described exercise would be a strategic plan of employing big data pattern(s) to achieve the business goals. Now the conversation can become more specific by identifying use cases within the organization to develop proof of concepts, a big data architecture to implement them, and to scale out to support the long-term strategy. After this point, solutions, vendors and technologies become relevant.
From experience with clients from all industries, e.g. finance, manufacturing, media, energy, e-commerce and more, we find that Hadoop often is the backbone of such an architecture although seldom the only technology required. Integrating it with other big data technologies and the existing enterprise architecture is a frequent requirement and demands a well-structured approach like the one outlined here to succeed.
This approach works equally well for small, medium and large organizations. Importantly, it gives clear guidance on the lowest level when developing proof of concepts and eventually delivering business-critical architectures and products, since it is clear how they tie into a big data strategy and how it helps achieve the business goals. This is essential, since the design of a big data enterprise architecture is a complex task. The technologies and patterns are constantly evolving and a significant amount of resources and time can be wasted. For example, choosing a suboptimal big data technology or pattern and forcing it on a use case is a subtle issue. It may only be discovered when a subsequent product has gone live and is faltering under scaling pressure or unable to accommodate new features.
A common example is choosing technology ad-hoc driven by developers’ preferences, hype or marketing. Unfortunately, the real issues often only show at scale or with increased complexity from future use cases. Proof of concepts as such can give false security without the experience and insight of how the demonstrated solution will hold up when scale out, high availability, guaranteed response time, future functional requirements, security, integration with other systems, compliance, governance, cost, support, and all the other real world issues become relevant.
In summary, big data success requires common sense and a structured approach. Do not get blinded by technology, hype or a “me too” frenzy. The steps are simple:
1. Business goals
They should be up to date and clear.
2. Discover your strategy
Discover how big data can support the business goals.
3. Discover your roadmap and use cases
Plan for the long term and identify short term use cases that are achievable.
Design an accompanying architecture and products.
4. Develop your use cases
Build proof of concepts to demonstrate value, viability, and solicit buy-in.
5. Deploy your architecture
Productionize your use cases with low risk, employing the learning from developing the proof of concepts.
6. Support and train
Focus on your business by employing services, vendors and support where feasible, and train your staff along the above mentioned path to bring them along with you on the big data journey.
We have employed this approach successfully with numerous clients at Big Data Partnership. Big data is here to stay and every organization should regularly evaluate it as part of its available strategies.