We are seeking an Engineering Director with deep Data Engineering expertise to lead the design and delivery of a next-generation AI-enabled data ingestion platform on AWS.
You will be #LI-hybrid based in Hyderabad and reporting to CTO.
This role owns the strategy architecture and execution for how data is ingested transformed modelled at scale across both batch and real-time (streaming) pipelines.
Data engineering is the core of this role.
You will define the pipeline architecture data models and quality frameworks that everything else depends on while modernising legacy and mainframe pipelines into cloud-native agent-assisted systems.
This is a hands-on leadership role that blends data engineering platform engineering and AI/DevOps excellence responsible for building high-performing teams and delivering reliable scalable and intelligent data systems.
Key Responsibilities
Data Engineering & Pipeline Architecture (Core)
- Own the end-to-end architecture for batch and streaming data ingestion source onboarding extraction transformation loading and serving.
- Design robust scalable data models and schemas (dimensional normalized and data-vault where appropriate) for a modern lakehouse.
- Build high-throughput fault-tolerant pipelines using distributed processing (Apache Spark / EMR Glue Flink) with idempotency replay and backfill built in.
- Architect real-time ingestion and change-data-capture (CDC) using Kafka / MSK / Kinesis and streaming frameworks.
- Standardize open table and file formats (Apache Iceberg / Delta / Hudi Parquet Avro) and manage schema evolution and partitioning strategies.
- Own pipeline orchestration and dependency management (Airflow / MWAA Step Functions dbt) with clear SLAs for freshness and latency.
- Drive performance scalability and cost optimization across compute and storage (partition pruning file compaction resource tuning FinOps).
Data Quality Governance & Reliability
- Embed data quality validation lineage and control frameworks directly into ingestion (Excellent Expectations / Deequ-style checks data contracts).
- Define and enforce data contracts cataloging and metadata management (AWS Glue Data Catalog Lake Formation).
- Ensure parity validation and controlled phased migration from legacy and mainframe systems with zero data loss.
- Implement observability for data pipelines freshness volume schema-drift and anomaly detection with SLOs/SLIs and error budgets.
- Ensure DevSecOps encryption access control and compliance (PII handling GDPR data residency)
across the platform.
Strategy & Engineering Leadership
- Define and execute the data-first AI-enabled ingestion platform strategy aligned to business outcomes.
- Build and lead teams spanning Data Engineering Platform/DevOps and AI/ML.
- Establish engineering standards reusable frameworks operating models and delivery roadmaps.
- Mentor senior and staff engineers; grow technical depth and set the bar for engineering craftsmanshi.
AI & Agentic Engineering
- Introduce AI-assisted engineering (Claude Code Copilot) across the data SDLC to accelerate pipeline development and testing.
- Leverage AWS Bedrock / LLM platforms for ingestion intelligence schema inference mapping validation and anomaly detection.
- Drive AIOps and agent-driven automation for auto-remediation intelligent alerting and self-healing pipelines.
Platform DevOps & Cloud Foundations
- Establish best-in-class CI/CD GitOps and release engineering for data pipelines and platform components.
- Standardize Infrastructure as Code (Terraform / CDK) and reusable self-service platform building blocks.
- Drive event-driven and microservices-based architectures on AWS-native services (S3 Glue Lambda EKS MSK/Kinesis).
Stakeholder & Delivery Management
- Partner with Product Architecture Data Governance and Business teams on roadmaps and prioritization.
- Manage delivery risk and cross-program dependencies; communicate trade-offs clearly to senior stakeholders.
Qualifications :
- 12 years in software/data engineering with 5 years in engineering leadership roles.
- Proven experience leading data engineering platform or data-intensive organizations at scale.
Data Engineering (Core)
- Deep expertise designing and operating large-scale batch and streaming data pipelines in production.
- Dstributed data processing: Apache Spark (EMR/Glue) and Flink or Beam.
- Data modeling and warehousing/lakehouse design (Redshift Snowflake Databricks Athena or equivalent).
- Lakehouse table formats and file formats: Iceberg / Delta / Hudi Parquet Avro; schema evolution and partitioning.
- Orchestration and transformation:
Airflow / MWAA Step Functions dbt.
- Streaming and CDC: Kafka / MSK Kinesis Debezium or equivalent.
- Expert-level SQL and Python (Scala/Java).
- Data quality lineage cataloging and governance frameworks (Great Expectations/Deequ Glue Catalog Lake Formation).
- Experience driving AI adoption in engineering/data workflows; familiarity with LLMs AWS Bedrock and AIOps platforms.
Observability Reliability & Security
- Prometheus Grafana OpenTelemetry CloudWatch; SRE principles and incident management.
- DevSecOps practices IAM encryption and compliance frameworks (PII GDPR data residency).
Additional Information :
Our uniqueness is that we celebrate yours. Experians culture and people are important differentiators. We take our people agenda very seriously and focus on what matters; DEI work/life balance development authenticity collaboration wellness reward & recognition volunteering... the list goes on. Experians people first approach is award-winning; Worlds Best Workplaces 2024 (Fortune Global Top 25) Great Place To Work in 24 countries and Glassdoor Best Places to Work 2024 to name a few. Check out Experian Life on social or our Careers Site and Glassdoor to understand why.
Experian is also proud to be an Equal Opportunity and Affirmative Action employer. If you have a disability or special need that requires accommodation please let us know at the earliest opportunity.
Benefits
Experian care for employees work life balance health safety and support of this endeavor we offer best-in-class family well-being benefits enhanced medical benefits and paid time off.
This is a hybrid remote/in-office role.
Recruitment Fraud Awareness - Experians recruitment process is conducted only through authorised channels. Recruitment communications will only be sent from an @ email address. Experian will never ask candidates to make any payment as part of an application interview assessment onboarding or recruitment process. To apply for roles or verify opportunities please visit Careers - Creating a better tomorrow together
Remote Work :
No
Employment Type :
Full-time
📌 Director of Engineering (Hyderabad)
🏢 Experian
📍 Hyderabad
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.