Job Description

We are seeking an Engineering Director with deep Data Engineering expertise to lead the design and delivery of a next-generation, AI-enabled data ingestion platform on AWS.

You will be #LI-hybrid based in Hyderabad and reporting to CTO.

This role owns the strategy, architecture, and execution for how data is ingested, transformed, modelled at scale — across both batch and real-time (streaming) pipelines.

Data engineering is the core of this role.

You will define the pipeline architecture, data models, and quality frameworks that everything else depends on, while modernising legacy and mainframe pipelines into cloud-native, agent-assisted systems.

This is a hands-on leadership role that blends data engineering, platform engineering, and AI/DevOps excellence, responsible for building high-performing teams and delivering reliable, scalable, and intelligent data systems.

Key Responsibilities

Data Engineering & Pipeline Architecture (Core)

- Own the end-to-end architecture for batch and streaming data ingestion — source onboarding, extraction, transformation, loading, and serving.
- Design robust, scalable data models and schemas (dimensional, normalized, and data-vault where appropriate) for a modern lakehouse.
- Build high-throughput, fault-tolerant pipelines using distributed processing (Apache Spark / EMR, Glue, Flink) with idempotency, replay, and backfill built in.
- Architect real-time ingestion and change-data-capture (CDC) using Kafka / MSK / Kinesis and streaming frameworks.
- Standardize open table and file formats (Apache Iceberg / Delta / Hudi, Parquet, Avro) and manage schema evolution and partitioning strategies.
- Own pipeline orchestration and dependency management (Airflow / MWAA, Step Functions, dbt) with transparent SLAs for freshness and latency.
- Drive performance, scalability, and cost optimization across compute and storage (partition pruning, file compaction, resource tuning, FinOps).

Data Quality, Governance & Reliability

- Embed data quality, validation,



lineage, and control frameworks directly into ingestion (Great Expectations / Deequ-style checks, data contracts).
- Define and enforce data contracts, cataloging, and metadata management (AWS Glue Data Catalog, Lake Formation).
- Ensure parity validation and controlled, phased migration from legacy and mainframe systems with zero data loss.
- Implement observability for data pipelines — freshness, volume, schema-drift, and anomaly detection — with SLOs/SLIs and error budgets.
- Ensure DevSecOps, encryption, access control, and compliance (PII handling, GDPR, data residency) across the platform.

Strategy & Engineering Leadership

- Define and execute the data-first, AI-enabled ingestion platform strategy aligned to business outcomes.
- Build and lead teams spanning Data Engineering, Platform/DevOps, and AI/ML.
- Establish engineering standards, reusable frameworks, operating models, and delivery roadmaps.
- Mentor senior and staff engineers; grow technical depth and set the bar for engineering craftsmanshi.

AI & Agentic Engineering

- Introduce AI-assisted engineering (Claude Code, Copilot) across the data SDLC to accelerate pipeline development and testing.
- Leverage AWS Bedrock / LLM platforms for ingestion intelligence — schema inference, mapping, validation, and anomaly detection.
- Drive AIOps and agent-driven automation for auto-remediation, intelligent alerting, and self-healing pipelines.

Platform, DevOps & Cloud Foundations

- Establish best-in-class CI/CD, GitOps, and release engineering for data pipelines and platform components.
- Standardize Infrastructure as Code (Terraform / CDK) and reusable, self-service platform building blocks.
- Drive event-driven and microservices-based architectures on AWS-native services (S3, Glue, Lambda, EKS, MSK/Kinesis).

Stakeholder & Delivery Management

- Partner with Product, Architecture, Data Governance, and Business teams on roadmaps and prioritization.
- Manage delivery, risk, and cross-program dependencies; communicate trade-offs clearly to senior stakeholders.

📌 Director of Engineering (Hyderabad)
🏢 Experian
📍 Hyderabad