When it comes to big data projects, it takes a variety of big data skills to deliver a project. Mastery of as many big data applications as you can help ensure the success of any big data project. The more you understand big data applications and how they deliver insight, the more value you will bring to the big data process.
Big data projects have many moving parts. Failure of any one of those components means the entire project will suffer. According to a Cap Gemini report, 60 percent of executives believe that big data applications will have a major impact on their industry, but only 27 percent say their big data projects have been successful. The primary reason cited is business culture rather than technology; data governance is lacking and entrenched legacy systems can’t support big data applications.
If you understand the interdependencies of big data applications, you will be in a better position to advise on business as well as technology decisions that can improve the chances for big data success.
Common Reasons for Big Data Failure
There are many variables that can affect the outcome of any big data project. Gartner’s big data expert, Svetlana Sicular, cites eight common reasons for big data failures:
Management resistance – Many managers still follow their gut rather than believing the data. Big data projects often fail because the outcomes are just ignored.
The wrong use case – A company can shoot too high and try to accomplish too much with a big data project, or they miss their target altogether. If the company is too ambitious, the big data project will fail. The other extreme is using big data to address the same use cases addressed using other technologies.
Asking the wrong question – Sometimes management uses big data to validate something they already know. They also phrase the question using the wrong assumptions, so they get flawed results.
Applying the wrong skills – Big data requires a diverse set of expertise and too often companies rely on the wrong resources. For example, if IT runs the project they lack the expertise to ask the right business questions, and you get the wrong results.
Problems beyond big data – Assuming you have all the components needed for big data applications there can be other points of failure, such as untrained personnel, network congestion, or some other external factor.
A disagreement on enterprise strategy – Big data applications succeed when they are at the core of company data operations. Too often, big data is seen as a peripheral project or an end in itself.
Data siloes – By their very nature, organizations tend to segregate data by department or function. Big data applications only work when they have access to all the data from all departments or business units for the desired insights.
Avoiding external problems – If you are concerned about the big data outcome you may not want to begin. In the case of pharmaceuticals, for example, running a sentiment analysis could trigger required reporting to the Food and Drug Administration.
Understanding Big Data Dependencies
With mastery of all big data applications and an understanding of the end-to-end process, you can avoid failure.
Big data is complex with multiple components. You have to identify the question to be addressed, assemble the data components, create an infrastructure to store and process the data, model the data, test and retest the results, present the findings, and then refresh and start again. These steps are interdependent and understanding those dependencies improves your chances of success. Even if your expertise is in one area, such as storage or analytics, knowing how those pieces contribute to the overall big data value chain is critical.
If you look at the eight common reasons for big data failure cited above, you can see how they are interdependent. For example, being able to archive and deliver stored data in a timely way to power analytics relies on interdependencies between infrastructure, database storage, understanding the nature of the data (e.g. structured or unstructured), developing the right algorithms, etc. Business professionals have to appreciate what the infrastructure can deliver and IT professionals have to appreciate the nuances of the business use case. Even if you have responsibility for only one big data component, understanding the end-to-end demands of big data applications goes a long way to ensuring overall success.