Big data has become an all-encompassing term that is being misapplied to refer to different kinds of data analysis. Understanding big data means understanding what makes big data analysis different from business analytics. You need to appreciate some key differences because understanding big data will make it easier to sell big data services.
Business intelligence (BI) has been around for some time; probably as long as there have been structured databases, and continues to be a growth market. Big data is still relatively new and goes beyond simple BI. Understanding big data starts with an appreciation that the data is truly big, too big to be handled using conventional BI tools, and it encompasses disparate kinds of data, both structured data in SQL and some related form, and unstructured data such as documents and graphics.
The market for cloud-based BI is growing at the same pace as big data. Forecasts show that the market for Advanced and Predictive Analytics is expected to grow from $2.2 billion in 2013 to $3.4 billion by 2018 at a CAGR of 9.9 percent. Cloud-based BI is expected to grow from $750 million to $2.94 billion during the same period at a CAGR of 31 percent. Big data hardware, software, and services, on the other hand, are expected to grow to $114 billion by 2018 at a CAGR of 30 percent. Clearly, there is still a lot of intelligence to be found in their data warehouses.
The Nature of Business Intelligence
What makes business intelligence valuable is it reveals a lot about operations by analyzing internal data. Performing sales projections based on previous quarters, ordering information, revenue from various products, and other data already stored in the data warehouse is part of BI.
Descriptive analytics use past events, such as sales performance statistics, to show what happened in the past. Using descriptive analytics, you can determine which specific stores sold more goods or which products produced more revenue.
Diagnostic analytics are for discovery using visualization and other diagnostic tools to determine why it happened. For example, using our sales analogy you can see that one store sold more of one specific product in December, which may indicate that product was popular because of Christmas.
Predicative analytics are what most managers mean when they talk about data analytics. Using statistical modeling and other tools you try to predict what will happen based on stored information. Using our sales example, predictive analytics would be applied to determine product shipments for the coming year.
Prescriptive analytics apply predictive models and repeatable processes to achieve desired outcomes. Just-in-time stocking strategies, for example, would use predictive modeling and current supply chain data to optimize inventory.
What Differentiates Big Data
Many wonder if big data is just a fancy term for business intelligence. Some of the techniques are the same, but the tools and data sources are different. What distinguishes big data from business intelligence are the three Vs:
Volume – The quantity of data to be analysed exceeds the capability of conventional analytics and statistical modeling tools. Big data analysis can scale to handle petabytes of data and opens up the possibility to handle volumes of data beyond what can be stored in most data warehouses.
Velocity – The speed at which data can be accessed is important. Most BI applications use historical data that dates back to previous weeks, months, or quarters. Big data often relies on real-time information to deliver rapid insights, such as real-time trading information. Insights, and the responses to those insights, are delivered in real time.
Variety – Big data looks at all sorts of information beyond what is structured and stored in the data warehouse. What makes big data so valuable is its ability to assimilate stored data and both unstructured and structured data feeds from external sources. For example, the success of a new product launch can use stocking data, sales data, and other internal data sources combined with Facebook data, Twitter feeds, online buzz, and other anecdotal, unstructured data.
Big data also requires different software tools, such as Hadoop and R, to analyze and model the data. Conventional statistical tools can’t accommodate the volume, velocity, or variety of big data.
The data center infrastructure to support big data also requires faster computer processing, more data storage, more memory, and more bandwidth. The drop in the cost for memory, storage, and bandwidth has put big data within economic reach of most organizations, but big data still requires a lot more computing resources than BI does.
So understanding big data is as much about understanding what it isn’t as what it is. If your customers want to take a closer look at their own data, that’s business intelligence. When they start looking outside their infrastructure at disparate, real-time data sources, that’s big data.