17 Nov
|
Nihilent
|
Kolkata
Senior Data Engineer Kafka & Streaming Data Processing
Location: Pune
Experience: 7-12 Years
About the Role:
We're hiring a Senior Data Engineer to build and scale real-time data processing pipelines using Apache Kafka and Debezium on AWS and open source stack like Apache Iceberg, Open metadata, Atlas, Apache Airflow. If you're passionate about streaming architectures and enjoy working across the AWS data ecosystem and open source stack, this role is for you.
Responsibilities:
Design & manage Kafka-based streaming pipelines (topics, partitioning, consumer groups), Flink
Design & manage Spark based Batch pipelines using Spark based framework
Configure and maintain Debezium connectors for real-time CDC
Build scalable ETL/ELT workflows using PySpark and AWS Airflow
Solid experience in utilizing Iceberg tables and AWS open source system.
Manage and optimize AWS Redshift for analytics & reporting
Ensure pipeline reliability, data quality, and performance at scale
Collaborate with cross-functional teams on data requirements
What We're Looking For:
10-12 years of Data Engineering experience with strong hands-on Kafka, CDC, AWS skills (must-have)
Experience with Debezium / CDC tools
Proficiency in PySpark, SQL, AWS Airflow, Glue, Iceberg and Redshift
Working knowledge of AWS services (S3, IAM, Lambda)
Strong SQL and data modeling fundamentals
Nice to Have:
Experience in the Finance/Banking domain
📌 Senior Data Engineer Kafka and Streaming Data Processing (Kolkata)
🏢 Nihilent
📍 Kolkata