Role Overview

The Data Engineer will work embedded within the data engineering function, partnering closely with the data platform team to support a large-scale data platform modernization effort.

- The role centers on migrating approximately 1,000 datasets to Snowflake, with Apache Airflow 3 handling orchestration; the core architecture and technical approach are already defined by the internal team, so the focus is on execution rather than design.
- Day-to-day work involves carrying out the migration, adapting existing pipelines to Snowflake, ensuring data quality and reliability, and supporting workflow scheduling and monitoring in Airflow.
- The overarching objective is to accelerate a set of time-sensitive migrations — Airflow 2 to 3, Confluent region migration, and the move to Iceberg — targeted for completion by the end of March 2027.

Key Responsibilities

- Migrate a set of pipelines from Apache Airflow 2 to Airflow 3.
- Migrate Confluent connections from the USE1 region to USE2.
- Migrate a set of pipelines from Hive to Iceberg using Snowpark Connect.
- Adapt existing data pipelines to Snowflake while ensuring data quality and reliability throughout.
- Support workflow scheduling and monitoring in Apache Airflow.

Must-Have Skills

- 5+ years of experience.




- Strong software engineering fundamentals.
- Hands-on Python development experience.
- Advanced SQL, including complex joins, CTEs, window functions, and performance tuning.
- Snowflake data modeling.
- Apache Airflow — developing and testing pipelines in Airflow and Python.
- Apache Kafka.
- Apache Spark, including hands-on PySpark experience.
- Iceberg and Parquet data formats.
- AWS experience, particularly S3 (data lakes, partitioning, and lifecycle policies).
- Testing strategy experience, with the ability to hands-on debug and test code.
- At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark.
- Solid communication skills.
- Ability to work independently.

Nice-to-Have Skills

- Hands-on experience with Snowflake migrations (data warehouse modernization, performance and cost optimization).
- Experience with Apache Airflow migrations (version upgrades, executor changes, or platform migrations).
- Experience using AI tools for development.

Pay: ₹500,000.00 - ₹1,500,000.00 per year

Benefits:

- Flexible schedule
- Paid sick time
- Paid time off

Work Location: Remote

📌 Sr. Data Engineer (India)
🏢 PeopleTree Knowledge Services
📍 India