By its nature, big data exceeds the capacity of most enterprise networks. The volume of data required for big data projects would cripple conventional network architectures, which is why big data projects use virtual storage and cloud-based resources. It also means that big data network monitoring is more crucial than ever. In order to maintain enterprise performance and security there have to be tools and procedures in place capable of inspecting and categorizing data traffic.
The three Vs of big data – volume, velocity, and variety – are all factors to consider in big data network monitoring. The volume of data inside and outside the firewall has to be monitored and managed. The variety of the data includes different data types, such as text, semi-structured data like social media, location-based data, and log-file data. And the velocity of data makes big data network monitoring more challenging, not only because data is being generated faster than ever before with digitized transactions, mobile user data, and machine data, but data has to be monitored in real-time.
IDC predicts that by 2020 there will be 450 billion Internet transactions every day. Today Walmart handles more than 1 million customer transactions each hour which generates 2.5 petabytes of data. YouTube generates 48 hours of new video content every minute. Facebook handles 100 terabytes of data each day. All this information has the potential to support big data analytics, but it will also have to be monitored and managed.
Applying Network Monitoring Switches
Big data network monitoring requires solutions that can operate in real-time, are scalable, that scan for malware, and that can facilitate data routing for analytics or storage. Big data architectures use distributed databases and computer processing, with large amounts of data spread across dynamically available servers to promote high availability and scalability. These factors make monitoring traffic more difficult.
The only way to get a complete picture for big data network monitoring is by using network monitoring switches. Forrester notes these switches are one of the three core technologies tools every big data center must have, and Gartner has labeled them “network packet brokers.” Network monitoring switches collect data from switched port analyzers (SPAN) and test access ports (TAP), copying all the data passing through the network to provide an overall picture of network traffic for security and performance optimization.
Applications for Big Data Network Monitoring
With network packet brokers in place, you now have the tools needed to take make the most of big data network monitoring for:
- Load Balancing: By having data flow through central enterprise intersections you can perform load balancing, managing real-time packet stream inputs and outputs for better throughput. Using centralized switches also ensures that session data stays together for better analysis.
- Data Filtering: The ability to filter inbound and outbound traffic makes it possible to focus on a specific packet stream or activity. For example, inbound filters can use packet headers to route and store data while protecting data in packet payloads. It also can eliminate unnecessary data streams to optimize bandwidth. Outbound filters can be used to prioritize data traffic for specific monitoring tools such as sending VoIP traffic to a VoIP analyzer.
- Real-time Data Analysis: As big data becomes more important to businesses to manage everything from manufacturing equipment to health monitoring, it’s going to be increasingly important to manage and filter data traffic in real time. Performing real-time analysis makes it possible to automate responses based on big data analytics, addressing issues faster than human response time. This is not only important for specific big data applications, but it can be used for enterprise security as well since big data can be used to identify viruses and network breaches before they occur.
- Managing Virtual Resources: Traditional network monitoring tools rely on a physical infrastructure. With big data you have to use virtual computers and data storage to accommodate scalable storage capacity and computing power. Big data network monitoring should include observing virtual resources, either using monitoring tools connected to a hypervisor, or by accessing packet streams to get more visibility into the virtual architecture. The objective is to make sure that the network infrastructure and applications are optimized to handle big data traffic.
Any network monitoring strategy should provide complete visibility into the network architecture. The factors that complicate big data network monitoring – the volume of data, the velocity of virtual storage and resources, and the variety of data types – makes monitoring data traffic more difficult, but with the right tools and management techniques monitoring big data traffic will pay off in better performance and data security.
Where do you see the greatest benefit from big data network monitoring?