Airflow Architect — Infrastructure Automation & Orchestration

Location: Chennai

Experience: 8–10 Years

Workplace: On-Premise Data Centre & DR

About the Role

We are looking for an experienced Airflow Architect to design, build and own an enterprise-grade Apache Airflow platform across on-premise Data Centre and DR environments.

The primary focus is infrastructure and operations automation, with data pipelines as a secondary workload.

You will be the technical authority for Airflow, owning architecture, HA/DR, security, performance, upgrades and the engineering team operating the platform.

Qualifications

Bachelor’s or Master’s degree in Computer Science, Engineering or equivalent.

Key Responsibilities

Architecture & Platform

- Design and implement highly available Airflow on-premise.
- Architect multi-scheduler, redundant web/API, Celery worker and RabbitMQ infrastructure.
- Design and test DC/DR architecture, including RTO/RPO, failover and failback.
- Establish Dev/UAT/Pre-Prod/Prod/DR environments.
- Own Terraform/Ansible-based Infrastructure as Code and CI/CD for DAG deployment.
- Integrate with LDAP/AD/SSO, Vault/CyberArk, PKI and enterprise storage/logging.

Infrastructure Automation

Build reusable Airflow workflows for:

- OS/middleware patching and rollback
- VM/server/container provisioning
- Backup and restore validation
- Certificate and credential rotation
- Health checks and auto-remediation
- Compliance and configuration-drift checks
- DR drills and evidence collection
- Access provisioning and recertification
- Cross-system runbook automation

Develop custom Operators, Hooks, Sensors and Plugins integrating with Ansible, SSH/WinRM, REST APIs, network, storage, backup and monitoring systems. Implement safe-automation controls such as approval gates, dry-runs, blast-radius limits, maintenance windows and audit trails.

MySQL Metadata Layer

- Architect and operate MySQL 8.x for Airflow.
- Experience with InnoDB, GTID, replication, InnoDB Cluster/Group Replication and PITR.




- Tune connection pools, InnoDB, queries and Airflow metadata tables.
- Design backup, restore, failover and schema-migration strategies.

Performance & Reliability

- Own Airflow capacity planning and performance tuning.
- Optimise schedulers, workers, concurrency, pools and queue performance.
- Implement monitoring using Prometheus/StatsD, Grafana, ELK/Loki.
- Monitor scheduler health, task latency, queue depth, DB connections and replication lag.

Security & Operations

- Own Airflow, Python dependency, OS, MySQL and RabbitMQ vulnerability management.
- Handle CVE assessment, Nessus/SonarQube findings and VAPT remediation.
- Implement TLS, RBAC, Vault-based secrets, network segmentation and privileged-access controls.
- Maintain SOPs, HLD/LLD, runbooks and operational documentation.

Leadership

- Lead a team of ~8 platform/support engineers.
- Conduct architecture and code reviews.
- Own production SLAs, incidents, changes and technical roadmap.
- Represent the platform in architecture and change-management forums.

Must Have Skills

- 8–10 years Infrastructure / Platform / DevOps / SRE experience.
- 5+ years production Apache Airflow experience.
- Proven experience building and operating Airflow on-premise, including HA and DR.
- Strong MySQL 8.x administration and performance tuning.
- Production experience with Celery Executor and RabbitMQ.
- Strong Python, Linux and shell scripting.
- Hands-on Ansible/AWX and Terraform.
- CI/CD experience with Jenkins, GitLab CI or equivalent.
- Experience with Grafana, Prometheus/StatsD and ELK/Loki.
- Strong understanding of enterprise security, vulnerability management and ITIL processes.
- Proven technical leadership and production SLA ownership.

Good to Have

- Control-M / Autosys / Tidal / Rundeck migration experience.
- ServiceNow / CMDB integration.
- VMware, Veeam, NetBackup or Commvault.
- Vault, Keycloak, LDAP/Kerberos.
- Apache Airflow open-source contributions.
- CKA/CKAD, RHCE, Ansible, MySQL DBA or security certifications.
- Astronomer Apache Airflow 3 certifications.
- BFSI, Telecom or Healthcare experience.

📌 Airflow Architect (Chennai)
🏢 TrizenAI
📍 Chennai