Senior Data Engineer - Responsibilities (7+ Years Exp)
Build scalable data ingestion pipelines for relational systems and unstructured/semi-structured sources.
Design and optimize lakehouse workloads using Apache Iceberg.
Optimize table layouts, partitioning strategies, compaction, and schema evolution for Iceberg-based datasets.
Implement best practices for versioning, time travel, incremental processing, and ACID guarantees.
Develop and tune Apache Spark (batch streaming) for performant, scalable data processing.
Work with AWS services (Glue, EMR, Lambda, Step Functions, S3) focusing on cost optimization and performance tuning.
Implement and manage Kafka and Kafka streaming pipelines for large-scale real-time processing.
Build and orchestrate data workflows using DBT and Airflow.
Leverage AI/LLM automation to accelerate ETL/ELT development, validation, and debugging.
Implement observability-monitoring, logging,



and alerting-to ensure platform stability and reliability.
Participate in code reviews to ensure code quality and share best practices with the team.

Required Skills
7 Years of relevant experience
Robust SQL, Python, and Spark experience
Deep experience ingesting and processing structured unstructured data
Solid knowledge of Apache Iceberg and lakehouse architectures
Hands-on AWS experience with tuning and cost optimization
Expertise with Kafka, Kafka Streaming, DBT, and Airflow
Solid understanding of distributed systems and data engineering best practices

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Senior Data Engineer Hyderabad
🏢 Milestone Technologies
📍 Hyderabad