When it comes to understanding big data you have to look behind the technology. The value of big data isn’t in the amount of data it can process, but in the insight that big data analytics can yield. Big data has value because it can assimilate vast amounts of different types of data. The three V’s – volume, velocity, variety – are what give big data its value, and what powers these three V’s is more and better data.
In 2015, it’s estimated that there will be 7.9 zetabytes of digital information available (which is the equivalent of 1,000,000,000,000 gigabytes). That number is expected to reach 35 zetabytes by 2020. It’s also expected that by 2020 data production will be 44 times greater than it was in 2009 and more than one-third of all data will pass through the cloud. Any of that data can be useful to some big data initiative; it’s all a matter of understanding big data and how to tap disparate sources to yield the desired insight.
Here are some of the market drivers that will shape our understanding for big data projects in 2015:
Tracking More of the Internet of Things
More devices are being manufactured with sensors that can deliver machine performance data over the Internet. The Internet of Things (IoT) is changing everything, including big data. The majority of data being used for big data analytics are tiny files of IoT sensor data, and IoT is much of what is driving the global data explosion.
IoT also can be one of the most valuable assets in big data. The value of the IoT market is expected to exceed $1.7 trillion in 2015, a 14 percent increase over 2014, and more than one third of spending on developing devices with embedded sensors will come from outside the IT and telecommunications industry.
As Intel explains, it will require three things for big data to tap the full potential of IoT:
- Intelligent systems that are capable of acquiring data and performing local data filtering and analytics;
- Smart systems that can provide interoperability between edge systems and federate data between edge systems and the cloud; and
- Smart analytics that assimilate data from across intelligent systems and distribute analytics at edge systems and inside the data center.
This kind of interconnected IoT framework is already at work, Factories, for example, are using sensor data to feed enterprise resource planning (ERP) and manufacturing execution systems (MES) that connect manufacturing and the supply chain to automate processes such as inventory ordering, distribution, and financial projections. Machine-to-machine (M2M) gateways are enabling automated data collection, and big data is using analytics to create intelligent processes that act on that data. With IoT machine logic can make decisions faster to deal with issues such as factory failures or a network security breach.
More Big Data in the Cloud
IT vendors are already starting to harness the cloud for Data-as-a-Service platforms, and vendors are merging cloud computing with big data analytics to generate insight from commercial and openly available data sets. Consumer sentiment analysis, for example, can be gleaned from social media and rich multimedia traffic; open data sources including everything from Facebook to YouTube.
Platform-as-a-Service (PaaS) offerings are growing in popularity to provide highly scalable, pay-as-you-go Hadoop clusters for batch analytics processing. Amazon, Google, IBM, Microsoft, and others are offering Hadoop in the cloud as prebuilt services and by 2016 more than half of the computing resources and 70 percent of data storage will be installed in these kinds of hyperscale data centers.
Cloud, mobile, and big data service providers will converge in the cloud and there will be new offerings coming in the near future.
The Next Step in the Evolution of Hadoop
Apache Hadoop is moving more into the mainstream with new trends that will affect the enterprise in 2015, and vendors scrambling and consolidating to take advantage.
Even though it is open source, Hadoop has matured to the point it is now a full-featured data platform serving as a distributed storage file system. Hadoop is becoming an enterprise data hub. Soon you will be able to perform different analytics and data manipulations simply by plugging them into the Hadoop framework.
Hadoop will start powering self-service big data analytics for both data scientists and business users. IT will no longer have to create centralized data structures. Hadoop will make it possible to move away from centralized storage to accommodate resources on demand, including new data sources. And the price of big data processing is going to drop as more vendors deploy enterprise-ready Hadoop platforms.
Understanding big data’s new market drivers will help you prepare to compete more effectively in 2015, and identify new potential sales opportunities. What are your big data clients clamoring for in 2015?