Understanding big data means appreciating that big data is defined as data sets too large to process using conventional database methods. You also probably know that what defines big data is volume (the size of the data pool), velocity (the speed at which data is processed, e.g. real time), variety (structured and unstructured data), and validity (the data is from a trusted source). Understanding big data also means you know that by analyzing all that data, it will reveal patterns that provide insight into a process, market trend, consumer attitudes, or some other business-critical question.
Big data is ideal for Information security. Identifying network security threats is largely about looking for data patterns that are out of the ordinary, whether it’s an unauthorized user from an unknown IP address or a denial of service (DOS) attack. Understanding big data techniques allows you to analyze network traffic to reveal anomalies that point to a data breach.
Network forensics is a subset of both information security and big data. Network forensics is the capture, recording, and analysis of network events to find the source of an attack to both prevent future attacks and perhaps for prosecution. Gartner analysts go further to define network forensics as having several functions, including:
- Full packet capture as part of digital evidence gathering
- Data retention for a period of time
- Access to captured data using search and other tools
- Packet header analysis
- Packet content analysis, including session viewing, application protocol analysis, file extraction, and the like.
Automating Incident Response
If information security is the objective, then network forensics can play an important role in revealing network attacks using big data tools and techniques for real-time analysis. The faster network threats can be identified, the less damage they can do. According to a study by the Ponemon Institute, a security incident detected within 60 seconds can reduce remediation costs by as much as 40 percent.
You can treat network forensics as another big data application. As with any big data project, network forensics requires a volume and variety of data, essentially all the network traffic. It calls for veracity, since the sources of the data need to be validated. And if network forensics are going to be used as part of information security (as opposed to just backtracking after a network breach), then big data velocity comes into play since the data has to be processed immediately to reveal patterns that point to potential threats.
Big data is an invaluable tool in digital investigation; however, its ideal application is in incident prevention. Network forensics can be used to uncover security issues, and big data analytics can be defined to not only detect security problems but also automatically address them by isolating a server, rerouting data traffic, or a host of other pre-programmed solutions.
Incident detection and response is an ongoing challenge for IT administrators. An Enterprise Strategy Group (ESG) survey of IT security professionals and CISOs revealed:
- 39 percent lack the staff for security operations and incident response teams.
- 35 percent are seeing too many false positive alerts.
- 29 percent are challenged by incident response requiring too many manual processes.
- 29 percent are hampered by a lack of integrated incident response tools, i.e. there are too many independent tools required.
Integrating network forensics with big data analytics can automate responses to security incidents and address all these concerns.
The Evolution of Forensics and Big Data
Of course, we won’t see self-securing networks overnight, but the potential is real. The Cloud Security Alliance (CSA) suggests the evolution of data analytics for in three stages:
- First generation is intrusion detection systems – Security architects increasingly understand that 100 percent protection is impossible, so they are working to implement layered security with reactive security and breach response.
- Second generation is Security Information and Event Management (SIEM) – SIEM systems aggregate intrusion detection sensors and apply rules-based responses to alarms. The objective is to detect with actionable intelligence.
- Third generation is adding big data tools to SIEM – This will shorten response time by correlating various security data sources using big data analytics, including correlating historical data for forensic purposes.
Network forensics can play a dual role in information security. It can provide evidence pointing to the source of a data breach after the fact, and it can be valuable in highlighting anomalies and pointing to a data breach in process, or at least soon enough to trigger a response.
Understanding big data, information security, and network forensics and how they can work together will present new sales possibilities. Security is always a primary concern, often with its own discrete budget, and could be the means to introduce the value of big data to a potential customer.