We are looking for an experienced Data Engineer with 6+ years of experience in designing, developing, deploying, monitoring, and optimizing scalable batch and streaming data pipelines. The ideal candidate should have strong hands-on expertise in Scala, Apache Spark, Kafka, Airflow, BigQuery, Dataproc, and GCP.

Key Responsibilities

Design, develop, deploy, monitor, and optimize ETL/ELT workflows for both batch and real-time/streaming data processing.
Develop scalable data processing solutions using Scala and Apache Spark.
Build and manage streaming pipelines using Kafka, including handling high-volume data and back-pressure scenarios.
Develop and maintain Airflow DAGs for complex data orchestration, dependency management, scheduling, retries, and idempotent execution.
Work extensively with Google Cloud Platform (GCP) services including BigQuery and Dataproc.
Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality validations, transformations, aggregations, and summarization.
Perform performance tuning and optimization of Spark jobs, Kafka pipelines,



BigQuery queries, and ETL workflows.
Implement monitoring and observability mechanisms to identify and troubleshoot pipeline failures, performance issues, and data-quality problems.
Create and maintain runbooks and operational documentation for production support and continuous optimization.
Troubleshoot runtime data issues independently and ensure timely resolution of production incidents.
Follow best practices for data governance, security, access control, and information security.

Required Skills

6+ years of experience as a Data Engineer.
Strong hands-on experience with Scala and Apache Spark.
Strong knowledge of Kafka and streaming data processing.
Experience with Apache Airflow and workflow orchestration.
Hands-on experience with Google Cloud Platform (GCP).
Robust experience with BigQuery and Dataproc.
Experience building both batch and streaming ETL pipelines.
Good understanding of

📌 Data Engineer – Scala | Kafka | Spark | GCP (Kolkata)
🏢 Zorba AI
📍 Kolkata