We are looking for an experienced Data Engineer with 6+ years of experience in designing, developing, deploying, monitoring, and optimizing scalable batch and streaming data pipelines. The ideal candidate should have strong hands-on expertise in Scala, Apache Spark, Kafka, Airflow, BigQuery, Dataproc, and GCP.

Key Responsibilities
- Design, develop, deploy, monitor, and optimize ETL/ELT workflows for both batch and real-time/streaming data processing.
- Develop scalable data processing solutions using Scala and Apache Spark.
- Build and manage streaming pipelines using Kafka, including handling high-volume data and back-pressure scenarios.
- Develop and maintain Airflow DAGs for complex data orchestration, dependency management, scheduling, retries, and idempotent execution.
- Work extensively with Google Cloud Platform (GCP) services including BigQuery and Dataproc.
- Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality validations, transformations, aggregations, and summarization.
- Perform performance tuning and optimization of Spark jobs,



Kafka pipelines, BigQuery queries, and ETL workflows.
- Implement monitoring and observability mechanisms to identify and troubleshoot pipeline failures, performance issues, and data-quality problems.
- Create and maintain runbooks and operational documentation for production support and continuous optimization.
- Troubleshoot runtime data issues independently and ensure timely resolution of production incidents.
- Follow best practices for data governance, security, access control, and information security.

Required Skills
- 6+ years of experience as a Data Engineer.
- Solid hands-on experience with Scala and Apache Spark.
- Strong knowledge of Kafka and streaming data processing.
- Experience with Apache Airflow and workflow orchestration.
- Hands-on experience with Google Cloud Platform (GCP).
- Strong experience with BigQuery and Dataproc.
- Experience building both batch and streaming ET

📌 Data Engineer – Scala | Kafka | Spark | GCP (Bengaluru)
🏢 Zorba Consulting India
📍 Bengaluru