Senior Data Engineer – Spark / Scala

Location: Bangalore, India

Industry: Retail & E-commerce

About the Role

We are looking for an experienced Senior Data Engineer to join our data engineering team supporting a large-scale US retail environment. The ideal candidate will have robust hands-on experience building and optimizing data platforms, developing high-volume data pipelines, and working with distributed data processing technologies.

This is a hands-on role for someone who can ramp up quickly, troubleshoot complex production issues, and independently drive technical deliverables from development through deployment.

Key Responsibilities

- Design, develop, and optimize large-scale data pipelines using Apache Spark and Scala.
- Build and maintain high-volume batch and streaming data pipelines.
- Develop data processing solutions using Spark, Hive, Kafka, and GCP services.
- Work with BigQuery, GCS, Dataproc, and Pub/Sub as part of cloud-based data platforms.
- Perform Spark job optimization, performance tuning, and troubleshooting.
- Design and maintain scalable ETL/ELT workflows and data pipelines.
- Troubleshoot complex data, application, performance, and production issues.
- Support cloud migration and modernization of legacy/on-premise data platforms.
- Develop reusable, scalable, maintainable, and production-ready data solutions.
- Implement and maintain CI/CD pipelines using Git, Jenkins, and related DevOps practices.
- Participate in code reviews, testing, production deployments, and operational support.
- Collaborate with engineering, analytics, product, and platform teams to deliver end-to-end data solutions.
- Analyze data quality issues and implement appropriate validation and reconciliation processes.




- Quickly understand existing systems and take ownership of assigned technical deliverables.

Required Qualifications
- 7+ years of experience in Data Engineering, Big Data, or related engineering roles.
- Strong hands-on experience with Apache Spark and Scala.
- Strong experience with Hive and distributed data processing.
- Hands-on experience with Kafka and streaming/event-driven architectures.
- Strong programming experience in Scala and/or Python.
- Proven experience designing and troubleshooting large-scale ETL/data pipelines.
- Strong understanding of distributed systems, data modeling, data quality, and performance optimization.
- Experience with at least one major cloud platform: GCP, AWS, or Azure.
- Experience with Git and CI/CD practices.
- Experience with workflow/orchestration tools such as Oozie or equivalent.
- Solid production troubleshooting and problem-solving skills.
- Ability to work independently in a fast-paced engineering environment.

Preferred Qualifications
- Strong experience with Google Cloud Platform (GCP).
- Experience with BigQuery, GCS, Dataproc, and Pub/Sub.
- Experience migrating on-premise Spark/Hive workloads to GCP.
- Experience with large-scale data migration, validation, and reconciliation.
- Knowledge of data quality, monitoring, and observability frameworks.
- Experience supporting complex retail, e-commerce, or high-volume transactional data environments.
- Experience working with multiple upstream and downstream data integrations.
- Experience with modern data engineering and cloud modernization initiatives.

Apply now or reach out with your resume to learn more about the opportunity at [email protected]

#Hiring #DataEngineer #SeniorDataEngineer #BigData #ApacheSpark #Scala #Kafka #Hive #GCP #BigQuery #Dataproc #DataEngineering #CloudComputing #ETL #Python #Jenkins #CICD #RetailTech #TechJobs #USJobs

📌 Senior Data Engineer (Bengaluru)
🏢 Andor Tech
📍 Bengaluru