What is Hadoop?
Hadoop developers design and create software applications that integrate with the Apache Software Foundation’s Java-based, open source Hadoop software platform, created in 2005. Development on the platform has continued over the last decade, and the latest version (version 2.4) was released in April 2014.
Hadoop is primarily used to distribute very large data sets across clusters of commodity servers, as opposed to relying on expensive supercomputers. It was designed to offer excellent scalability while providing robust fault tolerance designed to mitigate both hardware and software failures.
Hadoop’s key impact on the IT industry is its effect on the economics of mass parallel computing (often referred to as “Big Data”). Due to its open source licensing and the fact that it can be deployed using commodity hardware, Hadoop has positioned large scale computing at a much more affordable level for businesses and governments. Some of the companies that have implemented Hadoop-based solutions include Yahoo, Facebook, eBay and LinkedIn.
What do Hadoop developers do?
Hadoop developers create applications that tie into the Hadoop software framework. Many of these applications are based around querying massive data sets while providing the desired customization and security options found in conventional database user interfaces.
Other Hadoop applications provide traditional database monitoring and administration functionality. Apache also has other secondary Hadoop-related projects that may offer potential for additional new applications.
What’s the job outlook for Hadoop developers?
There are two main opportunities for Hadoop developers:
- Creating custom apps for employers/clients who have implemented Hadoop data clusters.
- Creating standalone, “off the shelf” apps for the Hadoop software platform.
In May 2013, CNNMoney listed “Hadoop and Java” as the number one tech skill desired by companies seeking Big Data employees. Similarly, in a 2012 news story, IT analysts at Gartner projected an increase of 1.9 million Big Data jobs in the U.S. between 2012 and 2015.
What’s the salary for Hadoop developers?
While there are no official salary numbers for Hadoop developers, the U.S. Bureau of Labor Statistics (BLS) does offer data on applications software developers. According to BLS data from May 2013, the median annual wage for applications software developers was $92,660, and the median annual wage for systems software developers was $101,410.
As Hadoop is a relatively new technology in terms of its industry adoption, most predictors agree that the demand for professionals with Hadoop and related data management and software programming experience will continue to grow over the next several years.
What training or certification do Hadoop developers need?
Hadoop application development is heavily based on the Java programming language, although they may also use other languages. Prospective developers can gain Java training through numerous organizations worldwide. Oracle, for example, offers two certifications for Java SE 7 programmers:
- Oracle Certified Associate
- Oracle Certified Professional
Additionally, Cloudera offers the Cloudera Certified Developer for Apache Hadoop (CCDH) designation.
Where are the jobs?
The BLS notes the five metropolitan areas with the highest employment per thousand jobs for application software developers, according to 2013 data:
- Washington: 18.39
- Virginia: 10.46
- California: 6.49
- New York: 4.59
- Texas: 3.99
Sources:
“Big Data could generation millions of new jobs,” CNN Money, May 21, 2013, Anne Fisher, http://management.fortune.cnn.com/2013/05/21/big-data-jobs-2/
“Gartner Says Big Data Creates Big Jobs: 4.4 Million IT Jobs Globally to Support Big Data By 2015,” Gartner, October 22, 2012, https://www.gartner.com/newsroom/id/2207915
“Software Developers,” Applications, Bureau of Labor Statistics, Occupational Employment and Wages, May 2013, http://www.bls.gov/OES/current/oes151132.htm
“Software Developers,” Bureau of Labor Statistics, Occupational Outlook Handbook, January 8, 2014, http://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm
“15-1132,” Bureau of Labor Statistics, May 27, 2014, http://www.bls.gov/oes/current/oes151132.htm
