As I discussed in my presentation at the Gartner Symposium/ITxpo in Florida, digital transformation is a key topic for business leaders today. While the impact of digital transformation is easily understood what is less clear are the steps to effectively pursue a digital transformation -- and the three keys to ensure successful digital transformation.
Disruption through data is significant competitive threat
Hotel chains have spent decades building their brand, developing properties, and improving their services, only to be completely disrupted by a competitor that doesn’t own a single hotel room. The largest hotel company in the world in terms of valuation and available rooms is Airbnb, with 1.5M listings, compared to the Marriott Starwood Group which has 1.1M rooms. Another example is Uber, which is now the largest car service without owning a single car or employing a single driver.
Leading organizations recognize that they need to take action. The Chairman, President & CEO of JPMC, Jamie Dimon, has declared “We will disrupt ourselves through data”. But what is the best approach to ensure success?
Source: IDC, Gartner, Analysis & Estimates: MapR
IT spending is at an inflection point, especially given the harsh realities that organizations face. Over the next four years, we’ll experience flat IT budget growth, but underneath that will be a steady decrease in legacy spend accompanied by a corresponding increase in next-gen technologies. The chart above also provides insight into the solution; the key to reducing costs while driving innovation is the data.
We are in the middle of a once-in-30-year re-platforming of the enterprise. The forecast also shows that within four years, 90% of data will be on next-gen technology. Data is the critical leverage point; it’s independent of the hardware choice. Not only does it work both in on-premise and cloud environments, it can intelligently handle the flow of data and processing across environments, including devices.
A disruptive data platform enables a wide range of processing and analytics, and supports running both new and existing applications. A disruptive platform can be used to dramatically lower costs. For example:
- IRI is a leading market research and information company that caters to 95% of Fortune global retailers & CPG companies. The company saved millions of dollars annually by offloading mainframe processing.
- Gracenote provides music and video metadata and technology to entertainment, automotive, and media companies. They offloaded their data warehouse processing to reduce costs by 75%, and decrease processing times from 24 hours to a few minutes.
- Experian is an international information services organization with global revenues close to $5B. They offloaded data from mainframe and SAN systems to increase performance at reduced costs.
How are these cost savings realized? The key driver is taking advantage of a much lower cost per TB of data. These companies are typically augmenting existing deployments and migrating data to reduce footprint and costs, with an operating expenditure savings of 80%, and a capital expenditure savings of 95%.
Data Warehouse Offload: Cost per Terabyte Comparisons
Here are a few examples of organizations that chose a big data platform to develop and deploy new applications:
- TransUnion developed a new self-service analytics platform, Prama, to give their customers better market insights to help them operationalize their decisions. Prama uses advanced technologies to harness massive underlying data assets and applies advanced analytics to give customers the ability to make better decisions, and to operationalize those decisions more effectively.
- United Healthcare provides health benefits to nearly 51M people. They generated a 2200% return on their big data platform with applications that included identifying and preventing fraud.
- Aadhaar is the biometric-based approach to authenticate identification in India. They estimate that they suffer from a 40% leakage in their government aid programs, and have attributed more than $1B in annual savings to the Aadhaar project. They perform over four million current transactions and resolve database lookups in less than 200 milliseconds. This example shows the need for scale, speed, and reliability in transformational applications.
These applications aren’t possible with traditional approaches. Application development is particularly difficult when it comes to bridging operational and analytical applications. IT must provide integration, but there are inherent barriers to integrating data, and these barriers result in delays, downtime, and prevent real-time operations.
Traditional approaches require an application-first approach. You start with the application and determine the data requirements. Then you prepare the data into specialized schemas to serve the application. With this approach you need to understand all the requirements so you can have the right data model. You have to extract the data, transform it, and get the data loaded.
This results in application silos. According to Gartner, a major data management issue facing organizations is the proliferation of data silos, with the typical organization having to deal with hundreds of separate data silos.
The Keys to Digital Transformation
1. The First Key to Digital Transformation: Data Convergence
This brings us to our first key to digital transformation – data convergence. The secret is to bring analytics and operations together. Convergence enables the immediacy of operational applications with the insights of analytical workloads. They leverage continuous analytics, automated actions, and rapid response to better impact business as it happens.
Historically, analytics and operations systems have been separated, with dedicated processes to extract, prepare, and load data, which introduces delays as well as administrative and security issues.
The benefits of a converged approach are to combine analytics and operations into a single platform. Not only does this combination drive efficiencies and reduce data duplication, but more importantly, it eliminates delays and latency, enabling real-time applications.
Data Platform Requirements
A converged data platform needs to scale, have features to ensure the data is available, protected, and supports legacy applications that can integrate with existing systems to reduce costs. Keep in mind that many applications require real-time integrated analytics. These applications are adjusting to impact business as it happens.
Comparing Data Platforms
Let’s look at how various data platforms stack up:
- Traditional storage lacks the required scale, cost effectiveness, and integrated analytics.
- Hadoop’s data layer has major limitations today, that existed when Hadoop was created 10 years ago. Hadoop stores data in the Linux file system, which was created in the early 90s. Hadoop lacks enterprise-grade features, including consistent snapshots and mirroring for DR. It’s a batch file system instead of real time, and legacy applications can’t use it like a standard file system.
- Spark has received a lot of attention and provides some excellent distributed processing and streaming analytic support, but it doesn’t have a persistence layer and there is no data platform, so it’s not a contender.
- Cassandra is a key value store. Because of the tradeoffs required with its eventual consistency model, you must select between scale, speed, and consistency, but not all simultaneously. In addition, Cassandra doesn’t support integrated analytics.
- Other data platform candidates include Kafka and MongoDB. These candidates might be a good choice for single workloads, but they lack the breadth of capabilities that are needed for a winning platform.
- MapR combines all the capabilities of the previous candidates into a single converged platform. The converged data platform supports operational and analytical applications at scale to impact business as it happens. Keep in mind that convergence is not simply about combining data sources and processing. There are many dimensions to convergence, including data-in-motion and data-at-rest.
Companies such as National Oilwell Varco, a $23B multinational company and a leading worldwide provider of oil equipment, components and services, is using the MapR platform to perform real-time analysis to optimize oil and gas drilling and production.
Security and fraud is another major area for real-time applications. American Express leverages big data to identify potential fraud when an American Express card is used anywhere in the world. Their platform protects $1 trillion in charge volume every year. Making decisions in less than 2 milliseconds, it supports approval of charges at the point of sale, with the least amount of disruption to customers.
2. The Second Key to Digital Transformation: Stream Processing
Harnessing these data flows and understanding their meaning and context are key building blocks for digital transformation and the development of breakthrough applications. These data sources can range from machine sensors, web events, biometric data, mobile, or other types of events.
Streams Enable Real-Time Applications
A stream is an unbounded sequence of events carried from a set of producers to a set of consumers. Producers and consumers don’t have to be aware of each other; instead, they participate in shared topics. This is called publish/subscribe. Streams can simplify the data integration pipeline, since a consumer can filter many producers’ topics, which are aggregated and then joined to the consolidated data.
Streams enable a range of real-time processing, from applications built around trending news feeds to operations dashboards, to real-time fraud detection and customized offers.
Customer Example: Liaison Transforms the Healthcare Ecosystem
Liaison Technology needed to make it easier for their platform to serve their health care customers that include hospitals, clinics, physicians, and payers. By using the MapR Converged Data Platform, the electronic medical record is a stream – and the stream itself is a system of record. Any updates are subscribed to and received by the various players and consumed as their application requires, whether as a search index or a database table, or a document file. This dramatically simplified the process and flow, and included integrated security for privacy and HIPAA requirements.
3. The Third Key to Digital Transformation: Application Agility
Streams enable event-based microservices, which are a necessary component for application agility, the third key to digital transformation. Microservices is an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.
Microservices, combined with Streams and a Converged Platform, provides:
- Application development across file, database, document and streaming services
- Ultra-scale, utility-grade performance
- Greater efficiency and simplicity than alternative architectures
- Integrated data-in-motion and data-at-rest and continuous and low latency processing
With a typical microservice architecture, a message broker communicates with the microservice but does nothing to coordinate the data flow. Until now, the architecture of a microservices application would rely on the use of multiple disparate platforms for supporting the different processing and message passing services. These different platforms would often correspond to physically separate clusters. Messages traveling between services would have to hop between platforms or cluster, adding latency and complexity. Developers and administrators need to keep track and manage the data flows and updates. This is relatively simple for ephemeral applications but with stateful applications that share data it can be a very difficult endeavor particularly when there are large volumes of data involved.
Converged Microservices Architecture
When you have an underlying fabric that integrates streams and a Converged Data Platform, this dramatically simplifies development, and makes real-time applications that rely on complex flow of data with automated responses and adjustments possible.
- For developers, a Converged Data Platform gives them tremendous flexibility and agility. Developers are free to choose the best approach for their project, whether it’s a complete application or a microservice.
- For architects and administrators, a Converged Data Platform simplifies administration, avoids cluster sprawl, and unifies security, data protection, and disaster recovery.
The road to digital transformation can be paved with a series of short-term projects that each generate positive ROI and payback. This series of tactical steps can result in a strategic architecture. In fact, MapR customers rapidly increase the number of applications running on a single cluster, with 18% of our customers running more than 50 applications on a single cluster.
Digital transformation and your ability to leverage data is at the core of your future competitiveness—it’s at the core of your ability to control costs and drive innovation. To successfully pursue digital transformation there are three keys to consider:
- Data Convergence
- Stream Processing
- Applications Agility
With the MapR Converged Data Platform, you can build innovative new converged applications that can transform your business by providing a competitive advantage that was simply not possible until now.