BIG DATA: The Next Frontier

You’ve heard the real estate mantra location, location, location – if Big Data has a mantra then it must go something like volume, variety, velocity. These three Vs are the characteristics that almost every IT expert agrees make up big data; that is data that is coming in in very large volumes, at very high speeds and in very disparate varieties and not forgetting from all manner of sources. 

But then big data isn’t just about the data itself but also the technologies, the processes, the solutions and the market dynamics that come into play whenever such data is encountered. Gartner, a leading IT research group puts this into perspective by defining big data as; “high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.” (gartner.com)

The reason why big data is such a big deal is because of its strategic and economic relevance to not only businesses, but to governments and consumers as well. Big data when properly mined and analyzed by any business can reveal patterns and help influence decisions that can help the business in enhancing productivity and thus giving it a competitive edge. For example a supermarket which through a loyalty card program are able to accumulate enough data on their customers can in turn be able to create accurate customer profiles from that data to gain deeper insight into their pool of customers and then from there come up with marketing and promotion campaigns that are accurately targeted.

Big data however brings in the challenge of data sets that are so large and/or too complex that they render traditional data management and analysis tools inadequate. For example let’s assume that the supermarket which typically has multiple points-of-sale and data entry personnel, also has a website, a twitter handle, a Facebook page, an SMS and WhatsApp line, and is collecting all the information coming in from all these sources and storing it somewhere. To be able to go through all these data and extract any meaningful and actionable insights from it, they will need to acquire the necessary resources that will enable them to do that accurately, efficiently and with speed. 

Typically, big data can be found in two formats; Structured that can fit in neat columns and can easily and quickly be organized and analyzed i.e. name, telephone, address; and Unstructured which is complicated and disparate and hard to make sense of i.e. social media feeds, emails, photos, video and audio files. According to experts, structured data only makes up about 20% of the total data out there, the rest (80%) is unstructured or a hybrid of both. Traditional data management tools were designed to work with structured data and though unstructured data has always been around these technologies did not support its analysis and organizations did not deem it important enough to do anything about especially bearing in mind the cost implications. 

Organization have since realized the potential gold mine within unstructured data and are now creating a demand for new capabilities, approaches and the right tools for collecting and analyzing big data. This has seen a series of data management waves over the past five decades that have brought us to the current big data era. A whole industry has now emerged and is now witnessing exponential growth as the demand by businesses looking to leverage on the big data wave grows. The International Data Corporation (IDC) - a US market research firm specializing in IT – back in 2010 predicted that the big data market will hit $ 16.9 Billion by 2015 but by 2014 they had adjusted that figure to 125 billion worldwide.