Airflow / Astronomer Platform Engineer (ETL SME)

JPID: DLTJP00063187 Budget: 25–26 LPA

Experience: 5–8 Years

Work Location: Hyderabad & Bangalore

Work Mode: Hybrid

Interview Mode: Virtual

Interview Rounds: 2 Internal Rounds

Work Timing: 24/7 Rotational; once in 4 months, night shift

Role Title

Airflow / Astronomer Platform Engineer (Contract / Short-Term Project — Migration Specialist)

Engagement Type

Small, fixed-scope project engagement (3 phases). Estimated duration: to be scoped per phase, but structured as a short-term/contract assignment rather than a permanent hire, given the finite, milestone-driven nature of the work.

Project Scope This Role Will Support

* Phase 1: Upgrade existing deployments from Astro Runtime 12.x → 13.8.0; validation, smoke testing, rollback readiness.

- Phase 2: Metadata database cleanup/retention (prerequisite for Airflow 3 migration).
- Phase 3: Migration to Airflow 3.x (Deployment Version 3.2-5), including DAG/plugin/operator/hook/API compatibility review and remediation.
- Phase 4: Post-upgrade validation and production readiness sign-off.

Key Responsibilities

* Execute Astro Runtime upgrade from 12.x to 13.8.0 across multiple deployments.
- Perform DAG and application validation, smoke testing, and operational verification after each upgrade step.
- Identify, validate, and run the Airflow metadata cleanup DAG; work with application teams to define data retention windows.
- Conduct a detailed compatibility assessment of all DAGs, custom plugins, operators, hooks, APIs, and integrations against Airflow 3 requirements.
- Remediate deprecated features and breaking changes identified during the Airflow 3 compatibility review.




- Execute the migration to Deployment Version 3.2-5 (Airflow 3.x) with rollback planning at each stage.
- Coordinate testing (unit/integration/regression) with application teams prior to production cutover.
- Document upgrade steps, issues encountered, and resolutions for audit/knowledge transfer.
- Support go/no-go decisions and production readiness checks.

Tech Stack

* Orchestration: Apache Airflow (2.x & 3.x), Astronomer / Astro (CLI, Runtime, Deployments)

- OS: Linux (RHEL/Ubuntu) — command line, shell scripting, service/process management
- Cloud: AWS (EC2, S3, RDS/Postgres, IAM, CloudWatch — as applicable to the workplace)
- Database: PostgreSQL (Airflow metadata DB — cleanup, retention, performance)
- Languages: Python (DAGs, plugins, hooks, operators), Bash
- Version Control: Git/GitHub or equivalent

Required Skills & Experience

* Hands-on experience upgrading/administering Apache Airflow (2.x, ideally exposure to 3.x) — DAG authoring, scheduler internals, operators/hooks/plugins.
- Direct experience with Astronomer / Astro platform — Astro CLI, Runtime version upgrades, Deployment management.
- Genuine, working-level Linux command-line proficiency — most upgrade, validation, rollback, and troubleshooting steps (log inspection, service checks, file/permission management, process management)



happen at the shell, so this should not be assumed as a byproduct of other skills.
- Comfortable operating in AWS-hosted environments (compute, storage, networking basics relevant to where Astro/Airflow is deployed).
- Experience troubleshooting and deploying via Kubernetes (pods, deployments, logs) since Astro runs on K8s under the hood.
- Solid Python for reviewing and remediating DAG, plugin, and hook code.
- Working knowledge of PostgreSQL for Airflow metadata cleanup, retention strategy, and performance.
- Familiarity with Airflow 2 → 3 migration specifics (breaking changes, deprecated APIs, DAG serialization changes, executor changes) — reference: Apache Airflow's official "Upgrading to Airflow 3" guide.
- Strong testing/validation discipline — smoke testing, regression testing, rollback procedures.
- Ability to work with distributed application teams to assess DAG-level impact.

Nice to Have

* Prior experience running a similar Airflow 2→3 or major-version Airflow migration in production.
- Experience with observability tooling for Airflow (logging, metrics, alerting).
- Familiarity with enterprise change management / rollback governance processes.

Deliverables

* Upgraded and validated Runtime 13.8.0 environments.
- Metadata cleanup completed and verified.
- DAG/plugin/operator compatibility assessment report.
- Migrated and validated Airflow 3.x (Deployment Version 3.2-5) environments.
- Post-migration validation and production readiness sign-off documentation.

Pay: Up to ₹2,600,000.00 per year

Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka

📌 Airflow / Astronomer Platform Engineer (ETL SME) (Bengaluru)
🏢 Clock Edge Technologies Pvt Lt
📍 Bengaluru