Spark Developer
Experience: 5–13 Years | Relevant
Location: Hyderabad
Job Overview:
We are looking for an experienced Spark Developer with 5–10 years of experience in building scalable data processing solutions using Apache Spark. The ideal candidate should have robust expertise in PySpark/Scala Spark, SQL, ETL development, and Big Data technologies, with experience in designing and optimizing high-performance data pipelines.
Key Responsibilities:
Design, develop, and maintain scalable data processing applications using Apache Spark.
Develop ETL/ELT pipelines for processing large volumes of structured and unstructured
data.
Write optimized Spark jobs using PySpark or Scala.
Work with distributed data processing frameworks and Big Data technologies.
Integrate data from multiple sources including databases, APIs, and cloud storage.
Optimize Spark jobs for performance, scalability, and reliability.
Collaborate with data engineers, analysts, and business teams to understand
requirements.
Perform code reviews, testing, debugging, and production support.
Ensure data quality, security,
and governance standards are followed.
Prepare technical documentation and support deployment activities.
Required Skills:
5–10 years of experience in Data Engineering or Big Data development.
Strong hands-on experience with Apache Spark.
Expertise in PySpark or Scala.
Strong SQL skills.
Experience with ETL/ELT development.
Knowledge of Hadoop ecosystem (Hive, HDFS, YARN).
Experience with Databricks is preferred.
Experience with Kafka or other streaming technologies is an advantage.
Familiarity with cloud platforms such as Azure, AWS, or GCP.
Experience with Git and CI/CD practices.
Strong analytical and problem-solving skills.
Preferred Skills:
Experience with Delta Lake, Snowflake, or Data Lake architectures.
Knowledge of Airflow or other workflow orchestration tools.
Experience in Agile/Scrum environments.
Good communication and stakeholder management skills.