Job Summary

We are looking for an experienced Data Engineer to join the Lakehouse Factory team, responsible for end-to-end migration of data stores from an on-premises Data Lake to an AWS-hosted Lakehouse environment.

This is a high-visibility, business-critical project involving pipeline migration, data transformation, consumption-pattern conversion, and rigorous data validation. The engineer will collaborate with internal data owners, platform teams, and stakeholders to ensure migrated data assets are accurate, reliable, and ready for production use.

Key Responsibilities1.

Data Pipeline

Migration

- Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
- Execute physical migration of datasets while maintaining data integrity and consistency.
- Coordinate with internal stakeholders and data owners to validate migrated assets and obtain sign-off.
- Consumption Pattern Migration
- Convert and optimize legacy SQL and Apache Spark-based consumption patterns for compatibility with Snowflake and Apache Iceberg.
- Analyze existing data usage patterns to identify and deliver the required data products.
- Collaborate with internal clients to understand business requirements and validate migrated datasets.
- Perform data reconciliation and quality checks to ensure functional equivalence with existing production flows.
- Data Validation and Reconciliation
- Develop and execute data validation and reconciliation processes to verify the accuracy and completeness of migrated data.
- Work with reconciliation frameworks to establish confidence in migrated datasets.
- Collaborate with the internal data management platform team and learn new workflows and language constructs as required.

Required Technical Skills1.

Basic

Qualifications

- Bachelor's or Master's degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.
- 3–5 years of hands-on coding experience in a collaborative,



team-based environment.
- Strong SQL troubleshooting skills and basic scripting experience.
- Skilled proficiency in Python or Java.
- Strong understanding of the Software Development Life Cycle (SDLC), CI/CD best practices, and Kubernetes (K8s) deployments.
- Core Data Engineering Competencies

Candidates should demonstrate a strong understanding of the following concepts:
- Temporal Data Modeling: Managing data changes over time, including Slowly Changing Dimension (SCD) Type 2.
- Schema Management: Schema evolution and enforcement strategies, particularly with Apache Iceberg.
- Performance Optimization: Data partitioning, clustering, and query performance optimization.
- Data Architecture: Normalization versus denormalization, and appropriate use of natural keys and surrogate keys.
- Data Reconciliation: Validating data correctness and ensuring functional equivalence between legacy and migrated datasets.
- Technical StackCategoryRequired TechnologiesProgramming & QueryingPython, Java, ANSI SQLData ProcessingApache SparkData Ingestion & IntegrationKafka, FTPData FormatsJSON, Avro, ParquetLegacy Data PlatformsHadoop, HDFS, Hive, Sybase IQTarget Data PlatformsSnowflake, Apache IcebergCloud & DeploymentAWS, CI/CD, Kubernetes (K8s)

Candidates are not expected to be experts in every technology; the team collectively needs coverage across the listed stack.

Core Competencies

- Demonstrates integrity, sound judgment, and ethical decision-making.
- Collaborates effectively with multiple teams and cross-functional stakeholders.
- Communicates clearly through concise written updates, structured verbal briefings, and proactive stakeholder engagement.
- Works effectively with global teams across different time zones and cultures.
- Takes ownership of deliverables and drives tasks through completion.
- Maintains urgency, quality, and professionalism while meeting deadlines.
- Demonstrates intellectual curiosity, raises risks early, and actively seeks feedback to improve delivery.

Skills: python,apache,sql

📌 Lakehouse Factory – Data Engineer (India)
🏢 Zorba AI
📍 India