Key Responsibilities

- Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
- Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
- Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
- Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
- Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
- Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
- Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.
- Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
- Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs.

Delta

Lake vs. Hudi), tooling, and cost optimization strategies.

- Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.
- Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

- 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.
- Deep, hands-on expertise with AWS data services: S3, Glue (PySpark ETL), Athena, Step Functions, Lambda, EventBridge, Lake Formation, SageMaker,



and CloudWatch.
- Production experience with Apache Iceberg (or Delta Lake / Hudi) table formats — compaction, snapshot management, schema evolution, and time travel.
- Strong PySpark and Python skills; ability to write, review, and optimize distributed data processing jobs at scale.
- Hands-on experience with CDC-based ingestion platforms (Fivetran, Debezium, or equivalent) across heterogeneous source systems.
- Proven experience designing and implementing medallion (Bronze/Silver/Gold) or equivalent multi-hop lakehouse architectures.
- Experience with data pipeline orchestration: AWS Step Functions, Apache Airflow, or equivalent; event-driven pipeline design patterns.
- Solid SQL skills;

experience with Athena, Trino, Presto, or equivalent query engines for large-scale analytical workloads.

- Familiarity with data governance tooling: catalog management (Glue Catalog, Apache Polaris/Iceberg REST), data lineage, access controls, and audit frameworks.
- Experience with Infrastructure as Code (Terraform or CloudFormation) for data platform provisioning and drift management.
- Solid understanding of dimensional modeling, schema design (star/snowflake), and data normalization for BI and analytics workloads.
- Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent professional experience.

Preferred Qualifications

- Experience operating Trino or PrestoDB on Kubernetes (EKS); tuning for sub-second query latency and multi-tenant workloads.
- Familiarity with streaming platforms (Kafka, Kinesis, or Pub/Sub) and real-time lakehouse patterns.
- Experience with Apache Polaris or other Iceberg REST catalog implementations.
- Exposure to SageMaker MLOps pipelines, Model Registry, and feature store patterns for ML data supply.
- Experience with dbt (data build tool) for SQL-layer transformation and documentation in lakehouse environments.
- Government contracting or ERP domain knowledge (Costpoint, Deltek, Oracle, or similar enterprise platforms) is a strong plus.
- AWS certifications: Data Engineer Associate, Solutions Architect Professional, or equivalent.

📌 Data Engineer (Bengaluru)
🏢 Deltek
📍 Bengaluru