To take on Big Data, business and technology leaders must:
- Develop an understanding of the specific skill requirements for the company's Big Data possibilities.
- Consider internal candidates for filling new positions.
- Ensure adequate training programs are in place.
- Set hiring goals with a view to creating a flexible analytics workgroup.
- Ensure that an awareness of analytics possibilities is spread throughout the firm.
Expectations for Big Data remain high. It offers businesses the real possibility of becoming more competitive in uncertain times. Yet it also embodies everything that is powerful and obscure in today’s world of information technology. It is seen as the top panacea, but it can also become a major headache.
In CIO Magazine’s 2014 Global CIO Survey, 72 percent of respondents ranked analytics and Big Data among their top technology initiatives. Companies are scrambling to incorporate it in their plans. Still, one thing that has consistently thwarted progress in the use of Big Data technology is a lack of people with Big Data skills. The sudden and growing importance of Big Data and Advanced Analytics has left many firms short of talent to implement these projects.
Skills requirements in this territory are complex. While much discussion has focused upon availability of “data scientists,” it is important to view Big Data as a team effort; moreover, an effort in which required skills will change and numerous participants will be needed.
Companies must determine their staffing needs thoughtfully. Flexibility is critical because technology is always changing. Senior data analysts and the ever-elusive “data scientists” are needed to lead the Big Data effort. These are specialists, often having postgraduate degrees. They have a wide understanding of the many components of analytic solutions, including business requirements, programming techniques, infrastructure and analytic demands. Data scientists are needed to understand the questions to ask of the data, to build the models and to propose a program for desired results. At a lower level, however, there is a much larger market for analyst and developer roles, and these roles are considerably easier to fill.
How to Build a Big Data Team
Building a Big Data team is likely to require a mixture of current staff, training programs, new hires, temporary workers, contractors, consultants, and, where possible, data scientists.
Specific technologies at present tend to focus around Hadoop and NoSQL, with Python being the coding language of choice. Skills in these areas are in increasing demand for all developers. Leading job site Indeed.com provides trends on these specific skills from its job listings. Limiting each technology by reference to “big data,” job postings over time show the recent and vary dramatic parallel acceleration in these listings. (See figure 1).
These skills are the easiest to track, but they also represent a range that includes specialty tools for specific analyses, additional programming languages and varying degrees of familiarity with infrastructure environments. It could change abruptly, of course—particularly as Hadoop moves to Hadoop 2.0 and new sub-specialties emerge. The graph clearly indicates the urgency of the requirement, which rose rapidly from 2011, plateaued during 2013, and in late 2014, went into overdrive as a new wave of investment in Big Data took hold.
Big Data is a team function that requires numerous skills in addition to technology. It demands skills in integrating analytic results with other technology; skills in explanation and skills in fitting results it in with existing practices. Teams also need to incorporate creativity, a deep understanding of the business and its environment, understanding of the strategic intentions of the firm, and, most importantly an ability to cooperate and collaborate within the team environment itself.
Train for Complex, Composite Skills
The employment market for Big Data will change as additional graduates become available; this is a high-profile skill that provides great rewards. It is something that all developers, contract workers and students can add to their available skills to improve job prospects. The market for Big Data analysts is likely to continue to grow over the next several years until supply catches up with demand.
Training for Big Data is also likely to become increasingly important. A February 2015 survey of IT leaders by staffing firm TEKsystems showed that 42 percent favored training as a leading way to address Big Data staffing shortfalls; 35 percent favored hiring contingent staff; and only 29 percent favored hiring full-time staff. However, the skills are more complex than those required for other disciplines; and the availability of outside training sources is relatively thin at present. Training firms are now moving in.
The need for complex, composite skills is fueling growth of corporate universities, which have been expanding rapidly for the past decade, and offer an increasingly flexible path for adding to available skills. Big Data has been accelerating this growth. Outside firms are also providing compound certification programs, and vendors are increasing their efforts in certification focusing upon skills that are necessary to use of their analytics products. Together, these developments ensure that training will become more readily available and can be obtained at a lower cost.
Technology to the Rescue
The difficulty in obtaining the skill sets required for Big Data is also fueling development of new tools targeted toward making Big Data processing available to less highly skilled individuals. The most recent entry in this area is IBM’s Watson Analytics, unveiled recently at the Vision 2015 conference. This tool, introduced initially for the finance industry, attempts to provide a simple querying system aligned with IBM’s sophisticated Cognos processing capabilities. It is designed to immediately respond to predictive analytics questions posed by a naïve user. It makes use of the Natural Language Processing (NLP) and query resolution skills demonstrated by Watson in winning the Jeopardy game show some years ago.
Automated tools for Big Data functions will change staffing requirements, taking the pressure off of some of the lower-level jobs and also making individual analysts more efficient. Along with this trend, we can see a continuing incursion of Big Data skills into the ordinary coding repertoire of developers. This is to be expected in the expanding technology skills marketplace. Ultimately, it will make these skills more common and reduce the hiring pressure.
It’s All About the Team
Overall, the struggle to develop and retain Big Data staff is likely to continue for some time. But it is critical to understand it as a team-building exercise. Staffing is not just about finding a key player—a data scientist. It is also about team building, spreading analytics knowledge throughout the firm, and incorporating new concepts and new tools within all IT departments. Skills are important, but the critical requirement is to create a flexible analytics workgroup capable of handling new Big Data initiatives as they develop.
If you can develop a flexible and highly skilled team that is capable of meeting today’s Big Data challenges, you will be a big step forward in matching the more complex requirements of tomorrow.
For an Infographic on Big Data skills, visit here.