About the Role:

We are looking for an experienced Senior Data Engineer who can quickly onboard and contribute to the design, development, optimization and support of large-scale data platforms. The ideal candidate should be hands-on, comfortable working with complex data pipelines and distributed systems, and capable of independently troubleshooting and delivering solutions in a fast-paced environment.

Key Responsibilities:

- Design, develop and optimize large-scale data pipelines using Apache Spark and Scala.
- Work with high-volume batch and streaming workloads across multiple data sources.
- Develop and support data processing solutions using Hive, Kafka and GCP services.
- Troubleshoot complex data, performance and production issues with minimal supervision.
- Contribute to cloud migration and platform modernization initiatives.
- Build and maintain ETL workflows and data orchestration pipelines.
- Perform performance tuning and optimization of Spark jobs and large datasets.
- Develop reusable, maintainable and production-ready data solutions.
- Implement and maintain CI/CD pipelines and engineering automation.
- Collaborate with engineering, product, analytics and platform teams to deliver end-to-end solutions.
- Participate in code reviews, testing, production deployments and operational support.

Technical Requirements:





- Strong hands-on expertise in Apache Spark and Scala for building and optimizing large-scale data pipelines.
- Experience working with high-volume batch and streaming workloads across multiple data sources.
- Strong experience with Hive and distributed data processing.
- Hands-on experience with Kafka and event/streaming-based architectures.
- Strong programming experience in Scala and/or Python.
- Experience designing, developing and troubleshooting large-scale ETL/data pipelines.
- Robust understanding of data modelling, data quality, performance tuning and distributed systems.
- Experience working with at least one major cloud platform - GCP, AWS or Azure.
- Experience with BigQuery and GCP services is preferred.
- Experience with ETL workflows and orchestration tools such as Oozie or equivalent.
- Hands-on experience with Git, Jenkins and CI/CD practices.

Preferred Qualifications:

- Strong Google Cloud Platform (GCP) experience.
- Exposure to BigQuery, Google Cloud Storage (GCS), Dataproc and Pub/Sub.
- Experience migrating on-premise Spark/Hive workloads to cloud platforms.
- Experience with large-scale data migration and reconciliation.
- Knowledge of data quality and observability frameworks.
- Experience working with multiple upstream and downstream integrations.
- Exposure to multiple cloud environments.

📌 Senior Data Engineer - Apache Spark (India)
🏢 SCALING THEORY TECHNOLOGIES
📍 India