Job Description

Job Purpose
Intercontinental Exchange, Inc. (ICE) is seeking an experienced Senior Data Engineer to join the AI Centre of Excellence (AI CoE) team. The AI CoE drives the adoption of artificial intelligence and advanced analytics across ICE's global exchange and financial data business, bringing together data engineering, machine learning, and applied research to deliver intelligent solutions at scale. This role sits at the foundation of that work, ensuring the data infrastructure, pipelines, and platforms the AI CoE depends on are robust, scalable, and fit for purpose.

The successful candidate will take ownership of the orchestration layer that keeps our data moving reliably, contribute to the lakehouse architecture that provides clean and trusted data to analytical and ML consumers, and develop the transformation logic that converts raw inputs into high-quality, governed data products. The role requires close collaboration with data scientists, ML engineers, and business analysts,



and carries meaningful input into the tooling decisions and engineering standards the team adopts.

We are looking for a candidate with demonstrable, production-scale experience in data engineering and pipeline orchestration. Technical depth is essential, as is the ability to produce well-structured, maintainable code and to operate effectively within a team-oriented engineering team.

Responsibilities
- Own Apache Airflow end-to-end. You will write DAGs that handle complex multi-step workflows across ingestion, transformation, validation, and delivery. That means getting the fundamentals right: idempotency, backfill safety, SLA alerting, dynamic task mapping, and thorough testing. You will also review other engineers' DAGs and raise the standard across the team.
- Keep Airflow running well on Kubernetes. You will manage the deployment using KubernetesExecutor or CeleryKubernetesExecutor, handle Helm upgrades, tune pod templates and resource limits, and deal with schedul

📌 Senior Developer, Data Engineer (Hyderabad)
🏢 Intercontinental Exchange Holdings
📍 Hyderabad