Senior Data Engineer Kafka & Streaming Data Processing

Location: Pune

Experience: 10-12 Years

About the Role:

We're hiring a Senior Data Engineer to build and scale real-time data processing pipelines using Apache Kafka and Debezium on AWS and open source stack like Apache Iceberg, Open metadata, Atlas, Apache Airflow. If you're passionate about streaming architectures and enjoy working across the AWS data ecosystem and open source stack, this role is for you.

Responsibilities:

Design & manage Kafka-based streaming pipelines (topics, partitioning, consumer groups), Flink

Design & manage Spark based Batch pipelines using Spark based framework

Configure and maintain Debezium connectors for real-time CDC

Build scalable ETL/ELT workflows using PySpark and AWS Airflow





Strong experience in utilizing Iceberg tables and AWS open source system.

Manage and optimize AWS Redshift for analytics & reporting

Ensure pipeline reliability, data quality, and performance at scale

Collaborate with cross-functional teams on data requirements

What We're Looking For:

10-12 years of Data Engineering experience with strong hands-on Kafka, CDC, AWS skills (must-have)

Experience with Debezium / CDC tools

Proficiency in PySpark, SQL, AWS Airflow, Glue, Iceberg and Redshift

Working knowledge of AWS services (S3, IAM, Lambda)

Robust SQL and data modeling fundamentals

Nice to Have:

Experience in the Finance/Banking domain

📌 Senior Data Engineer Kafka and Streaming Data Processing (Pune)
🏢 Nihilent
📍 Pune