Data center budgets are on the rise, but today’s data centers may not be ready to take on big data projects. The demands of big data can be beyond the capacity of many data centers, and part of big data consulting is assessing the demands of big data projects and whether it’s more cost-effective to build out the existing data center or outsource big data processing. Your big data consulting practice needs to be prepared to right size the data center and integrate more cloud resources.
According to IDC, the third platform is going to have a huge impact on new data center construction. IDC predicts that advent of cloud computing and other factors will slow data center growth, and the number of data centers will peak at 8.6 million in 2017.
Similarly, Gartner predicts that one of the disruptive forces in the data center market will be the dominance of the big cloud providers. Managed service providers (MSPs) are going to provide basic transport services where the big cloud providers will start delivering platform as a service (PaaS) and Infrastructure as a Service (IaaS).
With this migration to cloud services, many companies are going to look to big data consulting to show them how to keep sensitive information secure. They also are going to seek help from big data consulting to contain costs and balance in-house data center and cloud resources.
The Economics of the In-house Data Center
Just because the cloud is getting bigger doesn’t mean data centers will disappear. It does indicate that data centers will become leaner and more efficient. Experts predict that by 2018 more than 30 percent of all computing workloads will be running in the cloud.
Any company considering a big data initiative already has a data center in place. These organizations need to optimize their current data center investment while still maximizing their big data opportunity. Where these companies will add to their data center infrastructure are in areas that will pay off in applications beyond big data.
There are good reasons to keep data center operations in house. If companies are building an IT-based product, they need a secure, closed data center. Health care, financial services, government agencies, and other organizations that are subject to regulatory scrutiny and are going to want to keep tight control of their data and their security protocols for compliance. Other organizations are not going to trust cloud services with sensitive data, such as credit card information.
Companies are going to be looking to strike a balance between in-house data resources and cloud resources. Where the cloud can be of real value is providing extensible storage and data processing.
What to Outsource for Big Data
To accommodate the volume, velocity, and variety of big data requires both data storage and computing power. Any data architecture, including big data, is going to consist of data sources, data transformation, data processing, and data presentation. The challenge is determining which aspects of the architecture to keep in-house and which to host in the cloud.
The data sources for analysis make up the volume of big data. If the data is all in-house, then chances are you have a business intelligence project rather than a big data project. Big data benefits from pulling together internal and external data sources and analyzing them to reveal new patterns. That can mean storing terabytes or petabytes of data; more data than most data centers can handle. The proprietary data is usually stored in-house, which can mean the addition of more arrays of data storage, and the external data sources can be stored in the cloud.
Data transformation is the normalization of data for processing. To conserve bandwidth and speed analytics, much of this pre-processing is performed close to the data source. That means data transformation takes place both in the cloud and in the data center, depending on where the data is stored. Similarly, data processing or analytics can be performed both on premise and in the cloud, and the results delivered to the data center for presentation in a customized dashboard.
Virtualization plays a big role in big data. By abstracting the computing resources, including data storage, processing, applications, etc., it doesn’t matter where the data is stored or processed. Virtualization makes it possible to handle large volumes of data and compute-intensive applications using both in-house servers and resources and the more elastic cloud resources.
Big data will still demand in-house data center resources to handle sensitive, in-house information. However, to accommodate the big data projects in the future, existing data centers will become smaller and resources more concentrated, and more storage and processing will take advantage of the elastic, on-demand resources in the cloud. Big data consulting will help customers balance in-house and cloud resources.