SparkSpark Developer Job Description Template

Apache Spark has become one of the most used frameworks for distributed data processing. Its mature codebase, horizontal scalability, and resilience make it a great tool to process huge amounts of data.

Share

Apache Spark has become one of the most used frameworks for distributed data processing. Its mature codebase, horizontal scalability, and resilience make it a great tool to process huge amounts of data.

Spark’s great power and flexibility requires a developer that does not only know the Spark API well: They must also know about the pitfalls of distributed storage, how to structure a data processing pipeline that has to handle the 5V of Big Data—volume, velocity, variety, veracity, and value—and how to turn that into maintainable code.

Spark Developer - Job Description and Ad Template

Copy this template, and modify it as your own:

Copy to Clipboard

Company Introduction

{{ Write a short and catchy paragraph about your company. Make sure to provide information about the company’s culture, perks, and benefits. Mention office hours, remote working possibilities, and everything else that you think makes your company interesting. }}

Job Description

We are looking for a Spark developer who knows how to fully exploit the potential of our Spark cluster.

You will clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.

This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.

Responsibilities

  • Create Scala/Spark jobs for data transformation and aggregation
  • Produce unit tests for Spark transformations and helper methods
  • Write Scaladoc-style documentation with all code
  • Design data processing pipelines

Skills

  • Scala (with a focus on the functional programming paradigm)
  • Scalatest, JUnit, Mockito {{ , Embedded Cassandra }}
  • Apache Spark 2.x
  • {{ Apache Spark RDD API }}
  • {{ Apache Spark SQL DataFrame API }}
  • {{ Apache Spark MLlib API }}
  • {{ Apache Spark GraphX API }}
  • {{ Apache Spark Streaming API }}
  • Spark query tuning and performance optimization
  • SQL database integration {{ Microsoft, Oracle, Postgres, and/or MySQL }}
  • Experience working with {{ HDFS, S3, Cassandra, and/or DynamoDB }}
  • Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)
See also:Toptal’s growing, community-driven list of essential Spark interview questions

Find the right Spark interview questions

Read a list of great community-driven Spark interview questions.
Read them, comment on them, or even contribute your own.

Read the Questions

Hire a Top Spark Developer Now

Toptal is a marketplace for top Spark developers, engineers, programmers, coders, architects, and consultants. Top companies and start-ups choose Toptal Spark freelancers for their mission-critical software projects.

See Their Profiles

Steve Fox

Freelance Spark Developer

United StatesToptal Member Since July 25, 2019

Steve is a certified AWS solution architect professional with big data and machine learning speciality certifications. He has a diverse background, and experience architecting, building, and operating big data machine learning applications in AWS. Steve has held roles from technical contributor to CTO and CEO.

Show More

Luigi Crispo

Freelance Spark Developer

United Arab EmiratesToptal Member Since November 12, 2019

Luigi is a seasoned devops and leadership specialist with over two decades of professional experience in a variety of environments. He is passionate about technology and value-driven projects, and he is highly adaptable.

Show More

Tadej Slamic

Freelance Spark Developer

NorwayToptal Member Since May 6, 2019

With over a decade in the software industry, Tadej has helped startups launch their first product, assisted FTSE100 enterprises with digital transformation, been a part of the fintech boom, and helped particle accelerators cool down. He loves creating scalable back ends and is an expert in crafting modern and performant mobile, web, and desktop apps.

Show More

Andreas Bollig

Freelance Spark Developer

GermanyToptal Member Since October 9, 2019

With a Ph.D. in electrical engineering and extensive experience in building machine learning applications, Andreas spans the entire AI value chain, from use case identification and feasibility analysis to implementation of custom-made statistical models and applications. Throughout projects, he stays focused on solving the business problem at hand and creating value from data.

Show More

Mohammad Amin Khashkhashi Moghaddam

Freelance Spark Developer

SwitzerlandToptal Member Since October 1, 2019

Currently earning his master’s degree in computer science at ETH Zürich, Mohammad’s professional experience includes the technical management of a mobile advertisement product and working on products with tens of millions of users. He also has over two years of experience in data science and engineering—developing ETL pipelines, training, tuning big data infrastructures, and more.

Show More

Oleksii Sliusarenko

Freelance Spark Developer

UkraineToptal Member Since October 3, 2019

Oleksii is a senior research engineer specializing in machine learning with several years of hands-on, in-depth experience. In his free time, he competes in international programming and math competitions—and often wins. At Deloitte and Grammarly, he developed their core deep learning and AI algorithms. Oleksii has worked at all stages of R&D from problem formulation with clients to product deployment.

Show More

Sung Jun Kim

Freelance Spark Developer

AustraliaToptal Member Since March 11, 2019

As a highly effective technical leader with over 25 years of experience, Andrew specializes in data integration, data conversion, data engineering, ETL, big data architecture, data analytics, data visualization, data science, analytics platforms, and cloud architecture. He has an array of skills in building data platforms, analytic consulting, trend monitoring, data modeling, data governance, and machine learning.

Show More

Leonardo dos Santos Pinheiro

Freelance Spark Developer

AustraliaToptal Member Since July 15, 2016

Leonardo is a data scientist and machine learning engineer with eight years of industry experience across the government, energy markets, finance, and consulting sectors. He is well versed in work with both small and big data, specializing in the development and deployment of AI systems, and in the application of machine learning and optimization algorithms to generate predictive analytics and improve business process.

Show More

Weidong Ding

Freelance Spark Developer

CanadaToptal Member Since August 16, 2019

Weidong Ding has proven experience as a senior data/integration architect, recently focusing on SAP Data Services. He's detailed, hands-on, and efficient with comprehensive background planning, designing, and implementing information systems for leading organizations in the banking, transportation, retail, and government sectors. He leverages strong communication and customer service skills, working with clients and colleagues to achieve success.

Show More

Mark Perg

Freelance Spark Developer

IsraelToptal Member Since April 1, 2019

With a background from the Israeli elite intelligence unit and over 15 years of experience in software development and cybersecurity, Mark has everything in his skill set to deliver high-quality projects developed for your needs. As a full-stack engineer with experience in developing, architecting, and designing web applications with security and privacy by design approaches, Mark provides tailored high-quality solutions for your needs.

Show More

Levi Self

Freelance Spark Developer

United StatesToptal Member Since June 23, 2019

Levi has nearly a decade of experience in applied data science in a variety of industries with a concentration in the insurance industry. He's passionate about solving challenging problems that others find difficult or impossible. He's comfortable working independently and collaborating on teams. He is most at home in small startups with experience in enterprise as well.

Show More

Sign up now to see more profiles.

Start Hiring

Toptal Connects the Top 3% of Freelance Talent All Over The World.

Join the Toptal community.

By continuing to use this site you agree to our Cookie Policy.