About the Role

We are hiring two Advanced Software Architects to own the full lifecycle of our enterprise Data Lakehouse platform. These are senior, architecture-first roles we are looking for engineers who think in systems, make platform-level design decisions, and can build and operate a production-grade, multi-source lakehouse from the ground up, covering ingestion through consumption across a complex, multi-cloud source landscape.

You will be the technical authority for a platform that consolidates data from 18+ enterprise products (Costpoint, GovWin, Specpoint, Vantagepoint, and others) into a governed, medallion-architected data lake on AWS S3 with Apache Iceberg table format, orchestrated via AWS Step Functions, and queryable through AWS Athena and Trino. This role is end-to-end: you own ingestion, transformation, quality, orchestration, ML data supply, and BI consumption.

Platform Architecture You Will Own

Our current production stack — which you will architect, extend, and operate — spans the following layers:

Ingestion Layer

- Fivetran Managed Data Lake with CDC connectors; 13-minute sync frequency; Iceberg conversion and schema evolution across 500+ Bronze tables
- Multi-cloud source systems: OCI (Costpoint, Maconomy), Azure (Vantagepoint), AWS (GovWin) — all via CDC streams

Storage Layer (AWS S3 + Apache Iceberg)

- Bronze (s3://lakehouse-data/bronze/): Raw CDC data, Fivetran-managed, 500+ tables, 2-year retention
- Silver (s3://lakehouse-data/silver/): Clean, deduplicated, 100+ tables, 5-year retention — your team manages
- Gold (s3://lakehouse-data/gold/): Aggregated, business-optimized, 20+ tables, 7-year retention — your team manages

Processing Layer

- AWS Glue ETL (Serverless PySpark): bronze_to_silver, silver_to_gold, and metadata_capture jobs
- AWS SageMaker: Notebooks and endpoints (ml.c3.medium / ml.m5.xlarge) for ML model training, inference, and Model Registry

Orchestration & Automation

- AWS Step Functions: State machines for customer_360_pipeline, ml_training_pipeline, and metadata_capture
- EventBridge Scheduler for time-based triggers; Lambda functions for check_new_data, check_data_quality, check_model_drift, and send_alerts

Metadata & Governance

- AWS Glue Catalog (fivetran_bronze, lakehouse_silver, lakehouse_gold, lakehouse_metadata databases) — used by all services
- Fivetran REST Catalog (Apache Polaris): auto-generated, Bronze source of truth, read-only
- AWS Lake Formation: row-level security, column masking, and audit logging
- AWS CloudWatch: log aggregation, metrics & alerts, dashboards

Query & Consumption Layer

- AWS Athena (Phase 1): serverless SQL, $5/TB scanned




- Trino on EKS (Phase 2, optional): self-hosted, sub-second queries, 3–10 node cluster
- Consumption: ML Applications (churn alerts, customer insights, usage analytics), BI Dashboards (Tableau, Power BI, Looker), REST APIs (Lambda + API Gateway)

Key Responsibilities

- Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
- Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
- Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
- Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
- Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
- Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
- Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.
- Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
- Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.
- Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.
- Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

- 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.
- Deep, hands-on expertise with AWS data services: S3, Glue (PySpark ETL), Athena,



Step Functions, Lambda, EventBridge, Lake Formation, SageMaker, and CloudWatch.
- Production experience with Apache Iceberg (or Delta Lake / Hudi) table formats — compaction, snapshot management, schema evolution, and time travel.
- Strong PySpark and Python skills; ability to write, review, and optimize distributed data processing jobs at scale.
- Hands-on experience with CDC-based ingestion platforms (Fivetran, Debezium, or equivalent) across heterogeneous source systems.
- Proven experience designing and implementing medallion (Bronze/Silver/Gold) or equivalent multi-hop lakehouse architectures.
- Experience with data pipeline orchestration: AWS Step Functions, Apache Airflow, or equivalent; event-driven pipeline design patterns.
- Strong SQL skills; experience with Athena, Trino, Presto, or equivalent query engines for large-scale analytical workloads.
- Familiarity with data governance tooling: catalog management (Glue Catalog, Apache Polaris/Iceberg REST), data lineage, access controls, and audit frameworks.
- Experience with Infrastructure as Code (Terraform or CloudFormation) for data platform provisioning and drift management.
- Solid understanding of dimensional modeling, schema design (star/snowflake), and data normalization for BI and analytics workloads.
- Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent qualified experience.

Preferred Qualifications

- Experience operating Trino or PrestoDB on Kubernetes (EKS); tuning for sub-second query latency and multi-tenant workloads.
- Familiarity with streaming platforms (Kafka, Kinesis, or Pub/Sub) and real-time lakehouse patterns.
- Experience with Apache Polaris or other Iceberg REST catalog implementations.
- Exposure to SageMaker MLOps pipelines, Model Registry, and feature store patterns for ML data supply.
- Experience with dbt (data build tool) for SQL-layer transformation and documentation in lakehouse environments.
- Government contracting or ERP domain knowledge (Costpoint, Deltek, Oracle, or similar enterprise platforms) is a strong plus.
- AWS certifications: Data Engineer Associate, Solutions Architect Professional, or equivalent.

What You Will Build

You will be a founding architect of a strategic, cross-product data platform that serves 18+ enterprise applications and their analytics, ML, and AI workloads. The decisions you make on schema, storage format, query layer, governance, and orchestration will shape the data foundation of the company for years. This is a high-impact, high-ownership role with direct visibility to senior leadership.

📌 Advanced Software Architect | Data Engineer (Bengaluru)
🏢 Deltek
📍 Bengaluru