Big data development is one of the hottest hiring categories both because of the increase in demand and the scarcity of big data programming talent. The titles for these big data gurus includes big data scientist, big data software engineers, big data DBA/systems administrators, and anything else you can think of related to business intelligence, Hadoop, and NoSQL. And to take advantage of big data consulting means understanding the basics of big data application development
According to Dice.com, demand for NoSQL and Hadoop programming experts has risen 54 percent since 2013. In the category of NoSQL expertise, big data is specifically driving demand for experts in MongoDB, Couchbase, Redis, CouchDB, and other platforms. Big data also is driving demand for other types of programming such as Python, Java, and Ruby.
The value of big data consulting is in delivering business insight. The reason organizations are flocking to big data is because it can provide the insight to answer key business questions about pricing, shipping, customer preferences, business processes; business challenges that, if they can be optimized or overcome, translate into substantial revenue or savings. Big data consulting is about delivering reliable information to answer business questions. Your job as the big data consultant is to orchestrate the necessary processes to yield the result, so while you may not necessary need to be able to write the code, you do need to know enough about big data programming to assemble and analyze the necessary data resources.
The software development process for big data analytics isn’t very different from writing code for any other software application. The steps in the process are the same, with big data nuances.
- Big data requirements and data gathering – This is where your big data consulting expertise should shine. You need to be able to translate the business question into a big data problem. Big data consultants with a unique knowledge of the customer’s market have an advantage here since they can more readily identify factors that influence the question to be answered.
If your big data consulting practice focuses on retail, for example, then you have more expertise in identifying the factors affecting inventory stocking or pricing. This is the stage of the project where you assemble the stakeholders and define a big data use case, including identifying the kinds of insight needed, data sources available, and other factors that will affect the scope of the big data project.
- System analysis – Once you have a defined use case, you can start to blueprint the solution to address the use case. This is where a basic understanding of big data application development starts to become valuable. You need to identify available data sources and determine which sources are relevant; adding unnecessary data just slows the process. You also need to identify the enterprise resources needed in terms of enterprise storage, cloud storage, computational resources, etc., as well as the type of big data framework makes the most sense for the project, Hadoop, NoSQL, etc.
- System design – Now that you have a working blueprint of the big data use case it’s time to design the system. This is where a working knowledge of software development will help you keep the project on track. You need enough know-how to be able to work with the development team to design the analytics software with an eye toward delivering the insight specified within the big data use case.
- Coding – Next comes that actual coding. This is where the Hadoop or NoSQL experts go to work to build the analytical algorithm to process the big data sources to yield the desired results. While you don’t have to know how to write the actual code, to manage the project you should know enough to be able to allocate programming tasks and assess the outcome.
- Testing – Testing big data analytics is a critical part of the process. The most common approach is using an extract/transform/load approach that loads analytics into the data warehouse to identify inconsistencies, apply a common format, identify missing fields, and summarize detailed data. Effective testing is a matter of run and repeat to identify anomalies and eliminate bugs.
- Implementation – Once the analytics have been refined and tested, you are ready for the final implementation stage. An important extra step with big data is presentation; you need to deliver the results in a graphic form that can be readily interpreted. This is usually an R project or requires additional programming for graphic output.
As with most projects, you don’t need to know how to perform every task in the development cycle, but you do need to have a working understanding of software development as part of big data consulting. Building an effective and reliable analytic model is the key to big data success.