Big data projects require big data skills, but big data experts are hard to find. That’s largely because big data is a new field so the big data skills required for different projects are still ill-defined. There will be an ongoing demand for programmers, analysts, and managers with the big data skills needed to oversee big data projects because the supply of experts is still catching up with demand.
During 2016, Cisco, IBM and Oracle advertised 26,488 open positions requiring big data experience in the last 12 months. CIO.com reports that 40% of companies struggle to find needed talent. Why is there a big data skills gap? There are a number of contributing factors, including increasing demand for a new kind of technology skillset and demand for a new kind of project management expertise.
Hadoop, NoSQL, and Python programmers in demand
One of the most sought-after big data skills are software engineers, especially experts in Hadoop Python, and NoSQL.
Hadoop is still the dominant big data framework. According to Forbes.com, demand for Hadoop programmers is up 704%. Demand for NoSQL programmers—i.e., programmers with experience in unstructured data systems like MongoDB, is up 1,002%.
Python programmers, on the other hand, can write code for big data and other applications, such as cloud services. The demand for Python programmers is up 456% because of the simplicity and selection of data processing libraries, and because Python can be applied to web services and others elsewhere.
Growing demand for data scientists
The role of the data scientist is highly specialized. The data scientist usually has a background in statistics or mathematics. Their job is to be able to look at the big data findings and convert the data into business intelligence.
The formal training for a data scientist is similar to that of a data analyst, requiring computer science, modeling, statistics, analytics and math. What sets the best data scientists apart is their business savvy and their ability to communicate the big data findings to executives in a way that creates value for the organization. The other difference is that data scientists need to know how to assess all incoming data channels rather than just one data source. The data scientist can assess those channels from different perspectives to yield insight.
Demand for data scientists is at an all-time high. If you can’t find data scientists, many companies are training them. Some believe that being a big data scientist mostly requires mastery of a new set of analytics tools; if you have an expert with the right combination of computer science and business skills, they could assume the role of data scientist.
There are basic skills that a data scientist will need to have:
- Basic programming skills in R or Python
- Basic statistical skills
- Machine learning methods that can be applied, as needed, in lieu of R or Python
- Data normalization of “munging” in order to correct data values for consistency
- Data visualization, such as graphs and billboards, and communicating the findings
More project managers needed
One of the big data skills most often overlooked is project management. Someone has to have an overview of the entire project, the use case, objectives, data sets and outcome. One of the biggest reasons for big data failure is a lack of skilled oversight. The big data skills a project manager needs extend beyond the data center, and demand for information technology project managers continues to grow at a rate of 1.5 million opportunities per year.
A good big data project manager is part engineer, part salesperson, part diplomat, part educator and part systems planner. He or she is responsible for guiding the big data project from inception to completion, bringing together teams, managing workflow, assembling resources and advising on what will and will not work.
As with anything, hiring skilled big data professionals is a matter of supply and demand, and at present, the supply of skilled professionals is short. Many organizations are opting to develop their own in-house big data skills, tapping Java and SQL programmers to learn Hadoop, and looking for statisticians with business savvy. Of course, if you can’t hire the experts you can rent them.