Job Description – Spark + Scala + Python + GitHub Copilot
Position
Senior Data Engineer – Apache Spark / Scala / Python
Experience
4–8 years
Job Summary
We are looking for an experienced Data Engineer with strong expertise in Apache Spark, Scala, Python, and GitHub Copilot. The candidate will be responsible for developing scalable data processing solutions, building data pipelines, optimizing Spark workloads, and leveraging AI-assisted development tools to improve engineering productivity and code quality.
Key Responsibilities
Design, develop, and maintain scalable data processing pipelines using Apache Spark.
Develop high-performance Spark applications using Scala and Python (PySpark).
Build batch and near-real-time data processing solutions.
Perform data transformation, cleansing, aggregation, and enrichment using Spark.
Optimize Spark jobs for performance, scalability, memory utilization, and cost.
Work with large datasets across distributed data platforms.
Develop reusable and maintainable Scala/Python code following coding standards.
Troubleshoot Spark jobs,



performance issues, data-quality problems, and production failures.
Implement data validation, error handling, logging, and monitoring.
Work with cloud-based data platforms and distributed storage systems.
Use Git/GitHub for source control, branching, code reviews, and collaboration.
Leverage GitHub Copilot for code generation, refactoring, unit tests, documentation, SQL, and development productivity.
Review and validate Copilot-generated code for correctness, security, performance, and maintainability.
Collaborate with Data Architects, Data Engineers, Analysts, and business stakeholders.
Participate in Agile ceremonies, technical discussions, code reviews, and production support.
Required Skills
Strong hands-on experience with Apache Spark.
Strong programming experience in Scala.
Good hands-on experience with Python/PySpark.
Robust understanding of Spark SQL, DataFrames, and RDDs.
Experience with distribut

📌 Spark + Scala+ Python +Github Copilot (Bengaluru)
🏢 SPG Consulting
📍 Bengaluru