About the Role

As a Data Engineer , you will be responsible for designing, building, and maintaining the data pipelines and automation systems that power our cloud cost visibility, attribution, and performance insights. You will build scalable datasets and tooling that enable cost-aware decision-making across Product, Engineering, and Finance stakeholders, while optimizing data systems for performance, reliability, and cost efficiency. This role requires close partnership with Engineering and Finance teams to support cost optimization initiatives and ensure data quality across end-to-end workflows.

This position is open to candidates working in US-based timezones (EST/CST). Additionally, the Data Engineer will work embedded within the data engineering function, partnering closely with the data platform team to support a large-scale data platform modernization effort. The role centers on migrating approximately 1,000 datasets to Snowflake, with Apache Airflow 3 handling orchestration; the core architecture and technical approach are already defined by the internal team, so the focus is on execution rather than design.

Day-to-day work involves carrying out the migration, adapting existing pipelines to Snowflake, ensuring data quality and reliability, and supporting workflow scheduling and monitoring in Airflow. The overarching objective is to accelerate a set of time-sensitive migrations — Airflow 2 to 3, Confluent region migration, and the move to Iceberg.

Required Hours: Coverage until 12:00 PM PT.

Responsibilities

- What You'll Do Design, build, and maintain the Cloud Economics team's data pipelines and automation systems.
- Build and maintain scalable data pipelines and datasets that power cloud cost visibility, attribution, and performance insights.




- Design and implement data models and ingestion frameworks to support large-scale telemetry and usage data.
- Develop tooling that enables cost-aware decision-making across Product, Engineering, and Finance stakeholders.
- Optimize data systems for performance, reliability, and cost efficiency (query tuning, storage strategies, compute optimization).
- Partner with Engineering and Finance teams to support cost optimization initiatives and usage-based insights.
- Ensure data quality, validation, and auditability across end-to-end data workflows.
- Migrate a set of pipelines from Apache Airflow 2 to Airflow 3.
- Migrate Confluent connections from the USE1 region to USE2.
- Migrate a set of pipelines from Hive to Iceberg using Snowpark Connect.
- Adapt existing data pipelines to Snowflake while ensuring data quality and reliability throughout.
- Support workflow scheduling and monitoring in Apache Airflow.

Qualifications

- 2+ years of experience.

Required Skills

- Strong software engineering fundamentals.
- Hands-on Python development experience.
- Advanced SQL, including complex joins, CTEs, window functions, and performance tuning.
- Snowflake data modeling.
- Apache Airflow — developing and testing pipelines in Airflow and Python.
- Apache Kafka.
- Apache Spark, including hands-on PySpark experience.
- Iceberg and Parquet data formats.
- AWS experience, particularly S3 (data lakes, partitioning, and lifecycle policies).
- Testing strategy experience, with the ability to hands-on debug and test code.




- At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark.
- Strong communication skills.
- Ability to work independently.
- Strong Data Engineering foundation with proven experience building and maintaining end-to-end ELT/ETL pipelines in cloud environments.
- Solid SQL and Python skills (Scala is a plus).
- Experience with orchestration tools such as dbt, Airflow, or Dagster.
- AWS data ecosystem experience, including AWS Glue, Amazon Athena, and Amazon Aurora.
- Experience with Snowflake as a cloud data warehouse.
- Strong data modeling and large-scale data processing experience.
- Experience with CI/CD, version control (GitHub), and Infrastructure as Code (Terraform).
- Experience with REST API integrations.
- Demonstrated ability to optimize data pipelines for performance and cost.
- Strong analytical skills with experience in data validation, auditing, and troubleshooting production pipelines.
- FinOps Foundation Certification (Practitioner or Engineer). [Nice-to-have]

Preferred Skills

- Experience with cloud cost optimization, tagging strategies, or cost monitoring. [Nice-to-have]
- Experience with both AWS and GCP. [Nice-to-have]
- Experience with Datadog for monitoring and observability. [Nice-to-have]
- Background in data platform or shared infrastructure engineering. [Nice-to-have]
- Awareness or knowledge of IT security best practices as defined by ISO / SOC or similar standards. [Nice-to-have]
- Hands-on experience with Snowflake migrations (data warehouse modernization, performance and cost optimization). [Nice-to-have]
- Experience with Apache Airflow migrations (version upgrades, executor changes, or platform migrations). [Nice-to-have]
- Experience using AI tools for development. [Nice-to-have]

📌 Data Engineer (Kochi)
🏢 Astek
📍 Kochi