Key Responsibilities :

• Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
• Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
• Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
• Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
• Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
• Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
• Partner with data science teams on SageMaker data supply:



feature engineering pipelines, training dataset preparation, and model registry integration.
• Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
• Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.
• Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.
• Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

• 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.
• Deep, hands-on expertise with AWS data services: S3, Gl

📌 Data Engineer (Bengaluru)
🏢 Deltek
📍 Bengaluru