The MapR Hive ODBC Driver is a standard ODBC 3.52 driver, allowing our users to leverage hundreds of commercial and open source SQL-based tools, such as query builders and BI applications. For example, we’ve tested our Hive ODBC Connector with Excel, Tableau, MicroStrategy and a variety of 100% open source SQL tools, such as Kaimon (http://www.kaimon.cl).

As you may know, we deliver the most open Hadoop distribution, supporting the broadest ecosystem of applications. As a Hadoop distribution, we obviously support the Hadoop APIs, but we also support a standard NFS interface, which allows any file-based application to read and write data. With our new Hive ODBC Connector, we allow SQL-based applications to run SQL queries (via Hive).

Other Hadoop vendors have struggled to support connectivity with third-party SQL-based applications, resulting in specialized, proprietary connectors. For example, one Hadoop vendor had to build special-purpose connectors for Tableau and MicroStrategy (based on partial ODBC 2.x support), and even a rudimentary Web-based query builder. Another Hadoop vendor announced a specialized connector for Excel. We decided to take the more open approach of providing a single ODBC driver that complies with the latest ODBC 3.52 standard so that our users can utilize practically any SQL-based query builder or BI application with no special-purpose connectors.

Here’s a screenshot showing Kaimon, an open source SQL query builder, analyzing some Apache httpd logs via the MapR Hive ODBC Connector:

Here’s a similar screenshot with Microsoft Excel:

To get started, check out the Hive ODBC Connector page in our documentation: http://www.mapr.com/doc/display/MapR/Hive+ODBC+Connector

Only a few short years ago, the best tools available could only analyze small representative samples of big data sets. To figure out who will win the election, ask randomly selected people. To establish a correlation between behavior and outcome, do a study on a small group. This is an excellent way to find the common cases, but does nothing to turn up interesting anomalies–it’s like looking for a needle in a haystack by searching one percent of the hay–or one thousandth of one percent.

The new tools for analyzing Big Data can search the entire haystack–and are scaling up to deal with bigger and bigger haystacks. You may have heard this by now, but Big Data is big. How big? The Large Hadron Collider produces over a petabyte per month. The Square Kilometre Array radio telescope, which is being developed over the next decade or so, will produce at least an exabyte per day of astronomical data. According to Google CEO Eric Schmidt, every two days the world produces as much data as it did in total up until 2003. The amount of data produced by the world nearly doubles every year–and the world produced one zettabyte in 2010 (according to IDC).

It’s not just scientists, search engines, and social media sites that need to process this much data. Retailers, merchants, and credit card companies are looking for patterns in massive flows of transaction, click, and sentiment data to fine-tune marketing, prevent fraud, and optimize the customer experience. Here’s an example. Let’s say there’s a transaction on your credit card indicating that you bought a tank of gas. An hour or so later, another transaction shows that you ate lunch. If the restaurant is within 30 miles or so of the gas station, then the speed you would have had to travel–the card velocity–is a plausible 30 miles per hour. But if the restaurant is, say, 500 miles away, then the card velocity is a good indicator of fraud, unless you also own a jet pack. Only by processing data about every single transaction and its location can valuable insights like card velocity be gained.

The New York Times reported that data measurement could be as important as the invention of the microscope. By taking a snapshot of the data at various points in time, of course, you can tune the microscope by preserving a history of previous results, which makes it possible to further refine data analysis models.

This is the true beginning of the 21st Century. The possibilities that will grow from this new technology can barely be imagined today. So as you start to search for the needle in your haystack or even the needle in your jet pack, we’d love to hear the ways MapR made it possible for you.

In discussions with customers it is immediately clear that they are confused by the various descriptions of Hadoop High Availability. My personal favorite is another vendor’s description of their HA as providing “Hot Manual Failover”. Huh? How is a manual failover process “hot”?  What generates the heat exactly — the flames by business users when the cluster is unavailable? This has to be the biggest oxymoron to hit business continuity since “Highly Available – Not”. At least it’s clear from the latter that it isn’t really Highly Available.

In contrast, MapR has been designed specifically for High Availability and is the only Hadoop distribution with no single points of failure. Other distributions use a single NameNode and when that name node goes down, the entire cluster becomes unavailable and you lose data. With MapR, the NameNode function is distributed across the cluster. In a sense, MapR has a “No NameNode” architecture so there is no data loss or downtime, even in the face of multiple disk or node failures.

When we talk about high availability we’re talking about automated, stateful failover for all software and hardware errors. Automated re-replication of data means that your system will work through any errors without issues. MapR’s rolling upgrades guarantee high availability during routine hardware and software maintenance.

MapR is also built to give full data protection with Mirroring and Snapshots – features designed to efficiently maintain data integrity and business continuity across clusters and sites. This is significant because the replication that other Hadoop distributions use does not protect against user or application errors that are replicated across a cluster but with MapR you are fully protected. MapR makes data protection easy and built in. Furthermore, you will experience zero performance lost on writing to original during snapshot, a petabyte snapshot can be performed in only seconds.

So when considering High Availability for Hadoop make sure to get the complete picture, and then you can safely turn off the lights.

Cisco has published high performance reference configurations and Informatica announced a series of integration capabilities with MapR including real-time streaming into MapR with their Ultra Messaging technology.  InformationWeek states:  “For now, MapR has taken the lead in providing a low-latency option for streaming big data directly into Hadoop’s core MapReduce processing environment, and that counts as an edge on rival distributors.”

The timing of the partnerships and interest in MapR makes sense.  For customers that are considering distributions for Apache Hadoop, MapR provides unique functionality such as automated failover, dynamic file-based access and streaming support, and enterprise-grade data protection.  The value of the MapR distribution, is validated by the fact that the technology leaders in Storage (EMC), Data Management (Informatica), and Hardware (Cisco and others) are looking to MapR to help them address their customers Big Data needs.

Imagine trying to explain digital photography to someone 20 years ago. Film was more than adequate to the task at hand, very inexpensive, and easy to use. Sophisticated cameras took care of exposure and focus, abstracting away the “implementation details” and letting the photographer focus on snapping a picture. What motivation could the typical shutterbug possibly have for switching to digital photography? Pictures are pictures, after all.

Big Data is in the same state today. It’s difficult to imagine what benefits Hadoop can provide, in the same way that early adopters of digital photography could not have anticipated the impact of social media. Just as the digital revolution completely changed how we think about images, Hadoop is a complete paradigm shift in the way data is stored and analyzed. Hadoop is the key to a future we are only just beginning to imagine.

One of the first tasks of the MapR Academy is to get you started on the basics of this disruptive technology. As the MapR Academy grows, we’ll be adding more focused tutorials and advanced classes. I’d like to invite you to take a look around, watch a few videos, and let us know what you would like to see next in the curriculum.

Since Hadoop is a funny name and somewhat new to people they assume it must be risky. Huge amounts of investment and work have addressed these concerns. Hadoop has emerged as a standard. The rich ecosystem around Hadoop has provided a lot of flexibility, choice, and trained professionals. There are product-grade distributions available, (MapR) that provide full data protection, automatic stateful failover and business continuity. The deployed footprint, complementary products, and available technical resources all contribute to Hadoop adoption. And with that, the number and breadth of deployed Hadoop applications have expanded rapidly.

Another misconception about Hadoop, is that it is a batch process. This is an artifact of the HDFS implementation and not a limitation of Hadoop per se. MapR, for example, provides full support for streaming analytics and real-time processing.

Perhaps the biggest misconception is that Hadoop is a single, monolithic, component. Hadoop is a framework — a complete stack for distributing applications and data. Hadoop supports multiple programming paradigms and includes packages such as Pig, Hive and others. There are packages for data ingress/egress, ETL, and data integration, as well as specific components for machine learning. Most distributions integrate, test and harden these packages along with some proprietary extensions.

With respect to open source, the question about a distribution is not a simple binary “open” or “closed”. The question is what components are open and what areas do proprietary value-added components address. In the case of Cloudera, the proprietary extensions are in the management tools. MapR has chosen to innovate in the areas that provide the most benefits to customers while also being the most difficult for the community to effectively address. These also happen to be areas in which customers have the least desire to modify such as the underlying storage services. MapR’s distribution includes value-added improvements along with all of the open source programming, data access, programming, and machine learning packages.

These are some of the top misconceptions. Let me know what other areas you’d like us to address.

1. We want HA, but the NameNode is a single point of failure. This results in downtime due to hardware failures and user errors. In addition, it is often non-trivial to recover from a NameNode failure, so our Hadoop administrators always need to be on call.
2. We want to run Hadoop with 100% commodity hardware. To run HDFS in production and not lose all our data in the event of a power outage, HDFS requires us to deploy a commercial NAS to which the NameNode can write a copy of its edit log. In addition to the prohibitive cost of a commercial NAS, the entire cluster goes down any time the NAS is down, because the NameNode needs to hard-mount the NAS (for consistency reasons).
3. We need both a NameNode and a Secondary NameNode. We read some documentation that suggested purchasing higher-end servers for these roles (e.g., dual power supplies). We only have 20 nodes in the cluster, so this represents a 15-20% hardware cost overhead with no real value (i.e., it doesn’t contribute to the overall capacity or throughput of the cluster).
4. We have a significant number of files. Even though we have hundreds of nodes in the cluster, the NameNode keeps all its metadata in memory, so we are limited to a maximum of only 50-100M files in the entire cluster. While we can work around that by concatenating files into larger files, that adds tremendous complexity. (Imagine what it would be like if you had to start combining the documents on your laptop into zip files because there was a severe limit on how many files you could have.)
5. We have a relatively small cluster, with only 10 nodes. Due to the DataNode-NameNode block report mechanism, we cannot exceed 100-200K blocks (or files) per node, thereby limiting our 10-node cluster to less than 2M files. While we can work around that by concatenating files into larger files, that adds tremendous complexity.
6. We hired a new engineer who did not understand the architectural issues and ran a simple directory traversal (the equivalent of the find command). This created so much load on the NameNode that it simply crashed, and the entire cluster was down.
7. We need much higher performance when creating and processing a large number of files (especially small files). Hadoop is extremely slow.
8. We have had outages and latency spikes due to garbage collection on the NameNode. Although we are using the CMS (concurrent mark and sweep) garbage collector, the NameNode still freezes occasionally, causing the DataNodes to lose connectivity (i.e., become blacklisted).
9. When we change permissions on a file (chmod 400 foo), the changes do not affect existing clients who have already opened the file. We have no way of knowing who the clients are. It’s impossible to know when the permission changes would really become effective, if at all.
10. We have lost data due to various errors on the NameNode. In one case, the root partition ran out of space, and the NameNode crashed with a corrupted edit log.

When we looked at this list of NameNode-related problems, it was clear to all of us that the only viable solution was to eliminate the NameNode. Our engineering team spent two years re-architecting Hadoop’s storage layer (as well as advancing Hadoop’s MapReduce layer and developing the leading management suite for Hadoop).

The end result is that we have eliminated these 10 issues and many others. In my next blog post I’ll dive deeper into our no-NameNode architecture so that you can understand how it works, and why it really eliminates the issues with NameNode-based architectures (including all planned HDFS enhancements, such as HDFS Federation and HA NameNode). In the meantime, if you’ve run into other NameNode-related problems that I haven’t listed, let me know.

I believe another critical element is the commitment to partnering to achieve this growth if the foundational formula is in place. If you are a consulting partner, MapR will work with you to position your business advisory services, POC or Pilots, and other services. If you are a technology partner, we’ll work with you to differentiate our joint offering to customers. We do that with benchmarking, interoperability statements, reference architecture, joint promotions, event marketing, etc.

In the short time I have been with MapR, I’ve been impressed by the involvement of partners with our customers and MapR’s commitment to the channel. We want our partners to profit from services. The expertise in the Hadoop is finite today, but that is a big opportunity for our partners. The demand is there–we are seeing it from our customers. Our partners will profit from that demand on both the consulting and training front.

Joining MapR was an easy decision for me. If you are a consulting or technology partner, working with MapR should be an easy decision for you as well. I invite you to ask yourself a few simple questions: Who is the best company to partner with? Who provides the easiest to use and the most dependable product in the Big Data space? Who isn’t going to compete for training or consulting dollars in your accounts? Whose management team understands and is committed to partnering? The answer is MapR. To hear more, view my video and learn more about partnering with MapR.

Many enterprise IT vendors are wrapping themselves in some sort of “Big Data” cloak. The storage vendors talk about how they’ve always understood Big Data and cite the petabytes of storage that customers manage with their technology. The database and data warehouse providers make similar claims. Virtually, every vendor has some sort of Big Data presentation and by virtually every vendor I am of course including the virtualization providers. When MapR refers to Big Data we are talking about the MapReduce framework. This is the game changing approach popularized by Google.

We tend to take Google’s dominance for granted but when the Google search beta debuted in 1998 it was the 19th search engine on the market. The market was already well served with Yahoo, Excite, Infoseek, AltaVista, and a host of others. Within two short years, Google was the dominant player. The reason? MapReduce enabled Google to index much more data, much more quickly, and much more cheaply than any other provider. MapReduce is a paradigm shift, a new architecture that trumps existing approaches and provides any organization with the same power of changing their respective competitive landscapes. Google published a white paper on MapReduce in 2003. A Yahoo engineer, named Doug Cutting read the paper and the result was Hadoop. We’ve seen Hadoop emerge as a robust ecosystem with innovations happening across the Hadoop stack.

This is unfolding to be a big year. John’s predictions for 2012 encompass five major developments in Big Data. These include:
• Hadoop emerges as the safe platform choice for Big Data. The deployed footprint, complementary products, and available technical resources all reinforce the adoption of Hadoop.
• Real-time analytics take-off. Analyzing streaming data from application logs to messages augments existing batch applications.
• Hadoop applications move from experimental to mission critical. The number and breadth of deployed Hadoop applications also expands.
• Consulting firms augment their offerings with Hadoop specific consulting services expanding the number of available services vendors. Organizations benefit from the large and growing education and consultancy services.
• Big Data is no longer limited to companies that can ‘roll their own’ as the application ecosystem expands rapidly. In addition to the rapid expansion of Hadoop applications, 2012 sees the emergence of applications and services that leverage an underlying Hadoop engine.

We’re looking forward to a big year. We hope you join us.

• Ability to take advantage of next generation resource management framework: MapR users will be able to take advantage of MapReduce 2.0 once it is ready for production use. Although it is expected to take several months for the community to stabilize Hadoop 0.23, users will be able to take advantage of the combined benefits of MapReduce 2.0, such as backward-compatibility and scalability and MapR’s unique capabilities, such as HA (no lost tasks or jobs during a JobTracker or ApplicationMaster failure) and the high-performance shuffle.

• High-performance native access library: With Version 1.2, MapR provides a libhdfs implementation that bypasses Java altogether and provides high-performance access to the distributed file system from C/C++ applications and other compatible scripting languages. There is no need to recompile applications that use libhdfs, since the API (header file) is identical.

• Upgrade of various packages including HBase, Hive and Pig: The HBase package in the MapR distribution has been upgraded to release 0.90.4. In addition, MapR has identified several critical stability and data corruption issues in 0.90.4, which we have addressed by backporting 15 fixes from future HBase releases. Versions of Hive and Pig have also been upgraded in the MapR distribution, so users can leverage the latest bug fixes and features available from these Apache projects.

• MapR Virtual Machine (VM). MapR now provides a VMWare virtual machine that allows users to experiment with the MapR distribution. Although this environment is not suitable for any performance or scale testing, it makes it easy to experiment with some of MapR’s unique capabilities, such as NFS and snapshots. The VM is also a great asset if you are new to Hadoop, because you could be up and running on any environment (e.g., your laptop) within minutes.

• Additional performance improvements. The MapR distribution is already 2-5x faster than other distributions on typical Hadoop workloads, including the standard DFSIO and Terasort benchmarks, resulting in a significant hardware cost reduction. The 1.2 release continues to push the envelope, with a number of performance improvements in the platform (file system and MapReduce layers).