Pontis Transitions From a Legacy Architecture to the MapR Distribution including Apache Hadoop

Pontis chose MapR as their target architecture to integrate Hadoop into their solution, and built a real-time operational analytics environment on Hadoop. By gradually transitioning from their legacy architecture to MapR, Pontis achieved higher scalability and lower TCO in incremental steps. While the previous system hit the wall at 100 million transactions a day, Pontis is now able to engage with 400 million customers on a daily basis.

The Business

Pontis is a developer of online marketing automation solutions for the telecommunications industry. Pontis solutions replace traditional, mass segmentation telco tactics with multi-dimensional context marketing that lets communications service providers build continuous, on-going dialogs with individual customers and respond to personal needs and preferences. Their current clients include leading global telco companies such as Vodafone, Telefónica, and VimpelCom.

The Challenge

In 2011, Pontis realized their relational database management systems architecture was becoming outdated. The daily transactions had rapidly approached the architecture’s full capacity—about 100 million transactions a day. The architecture could technically grow, but at a steep cost. At the same time, Pontis started to acquire new deals with even bigger clients whose initial demands reached billions of daily transactions.

Pontis decided to look into big data technologies. They needed a new architecture that could handle their increasing scale, support clients who would upgrade to big data, and support clients who didn’t need to upgrade.

MapR Solution

Pontis wanted an enterprise-grade Hadoop distribution that could run a mission critical telecommunications system. They needed features such as high availability (no single point of failure), backups and snapshots, consistent response time for sub-second OLTP responses, and support for heterogeneous hardware. They evaluated three Hadoop distributions on capabilities such as performance, high availability, and manageability. After a thorough evaluation process, they chose MapR. They then embarked on a five-phase project to transition from their legacy architecture to Hadoop. Their MapR solution offered multiple benefits along the way.

Results

Increasing scale The move to MapR greatly increased the company’s ability to scale. Pontis’ legacy architecture used Oracle for OLTP and SQL Server for analytics. When they transferred Oracle queues to the MapR file system, it required no change to business logic code, and allowed Pontis to nearly double their OLTP system scale. At the end of the migration, Pontis had moved from managing about 100 million of transactions a day to billions per day.

NFS enables smooth, fast data transfer NFS streamlined the data transfer process. When Pontis offloaded the analytics activity from SQL Server, they transferred the data warehouse to Hive, and moved the complex analytics calculations to MapReduce. They were able to load new imports and logs via NFS. This avoided the extra work required to implement other loading technologies like Kafka and Flume. The logs and additional client files were terabytesized Avro files that include a complex structure of individual subscriber history, with thousands of objects and all of the subscribers’ activities and aggregations over the past six months. After each run, they’d generate a new profile for the next day’s run, enabling good overall performance. They used Sqoop to move individual subscriber information data back into SQL Server as the profile store to avoid having to change working client code. This was very fast, and they were able to insert tens of millions of records in minutes.

Improved analytics capabilities The MapR cluster gave Pontis a tremendous increase in their analytics capabilities, and the architecture could theoretically grow to support an unlimited number of subscribers and customers. They gained higher capacity, simplified the architecture, and maintained high availability with no data loss.

They could now collect two or three months of history data, which they couldn’t do before. And by using MapReduce, the subscriber profile calculation only took two or three hours, and could be improved by simply adding servers to the cluster.

Pontis also gained from the MapR ability to store many small files. Pontis last counted 130 million files, which helps them to avoid continual file concatenation techniques required with other distributions.

MapR-DB increases capacity to billions of transactions per day Pontis saw a major increase in capacity when it moved its data to MapR-DB. When Pontis communicates with clients, each of these interactions is documented in an activity log stored in tables. These tables can reach billions of lines in a relational structure. With the move to MapR-DB, Pontis now has one line per customer, with thousands of columns describing that customer’s transactions. This allowed some of their OLTP activity to run through an additional channel, which further increased their system capacity.

All Oracle functionality was also moved to MapR-DB, so the OLTP subsystem could reach unlimited capacity in terms of customers and events per day, contingent on the number of servers. Pontis can now handle hundreds of millions of subscribers, and billions of transactions per day.

Cost savings By removing Oracle and SQL Server from Pontis’ architecture and adopting MapR, the company realized a significant positive impact in terms of system cost, deployment, and monitoring. With MapR, they consolidated disparate systems to create a realtime operational analytics deployment.



DOWNLOAD PDF