- Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala.

- Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing.

- Design and implement data models including Dimension and Fact tables, with clearly defined grain, relationships, and measures.

- Implement Slowly Changing Dimensions (SCD) and appropriate incremental processing strategies.

- Work with AWS data engineering and serverless services such as Glue, S3, Lambda, EMR, and Step Functions.

- Develop and maintain reliable, scalable data pipelines with appropriate error handling, logging, monitoring, and data quality checks.

- Apply data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.

- Work with Apache Iceberg, including table design, partitioning, schema evolution, and incremental processing.

- Develop data transformation workflows using DBT and orchestrate pipelines using Airflow.

- Troubleshoot and optimize data pipelines for performance, scalability, and cost efficiency.

- Participate in code reviews, testing, debugging, and continuous improvement of data engineering solutions.

Required Skills





- 4–7 years of relevant Data Engineering experience.

- Strong SQL skills — SQL is a core requirement.

- Strong hands-on experience with Apache Spark and Python or Scala.

- Hands-on experience with AWS data engineering and serverless services, particularly AWS Glue and S3.

- Valuable understanding of Data Modelling, including:

- Dimension and Fact table design

- Defining appropriate Fact grain
- Dimension-to-Fact relationships

- Implementing SCD Type 1 / Type 2 and other appropriate SCD patterns

- Strong understanding of data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.

- Good understanding of distributed data processing and Spark performance optimization.

Additional Responsibilities / Good to Have

- Experience with Apache Iceberg and lakehouse architectures.

- Experience with DBT for data transformation and modelling.

- Experience with Airflow for workflow orchestration.

- Kafka / Kafka Streaming experience is a good to have.

- Exposure to AI/LLM tools for development, debugging, testing, and documentation.

📌 Data Engineer (Hyderabad)
🏢 Milestone Technologies
📍 Hyderabad