– Senior Data Engineer / Platform Re-Engineering Lead (Azure Synapse & Databricks Migration)

Role Overview We are looking for a highly experienced Senior Data Engineer to lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics. The role requires deep technical seniority to audit, understand, and validate a complex end-to-end data architecture spanning source ingestion through to consumption — and to drive a future migration of validated workloads to Databricks.

This is not a greenfield role: it demands the ability to reverse-engineer existing implementations, assess their correctness, and own the technical migration strategy.

Key Responsibilities

- Lead the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
- Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
- Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
- Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
- Design and implement target-state architectures aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
- Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
- Define and execute migration strategies for existing workloads to modern data platforms, preserving existing governance and control framework semantics
- Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
- Collaborate with business, data governance,



and architecture stakeholders to validate embedded business rules and data quality requirements
- Provide technical leadership across re-engineering and migration workstreams, contributing to decommission planning for legacy components

Core Technical Skills

- Azure Synapse & Data Platform Mandatory hands-on expertise with:
- Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool)
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Delta Lake on Azure (Synapse Lakehouse patterns)
- Oracle Golden Gate Replication for real-time source integration
- Azure Analysis Services and Power BI consumption layer patterns
- Deep understanding of medallion architecture: Raw / Harmonized / Conformed / Consumption layers
- Robust knowledge of SCD Type 0/1/2, CDC patterns, soft/hard delete, and retroactive change processing
- Experience with Synapse SQL Pool — stored procedures, control tables, and data quality validation patterns
- Experience with audit, balance, and control frameworks — parameterized, modular pipeline governance at enterprise scale
- Familiarity with config-driven and automation-first pipeline patterns (YAML, PySpark, SQL-driven generation from mapping documents)

Databricks & Lakehouse

- Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred)
- Strong Apache Spark skills (PySpark / Spark SQL)
- Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse
- Ability to re-implement governance and control frameworks natively in Databricks (audit logging, reconciliation, DQ checks)
- Experience with Delta Lake features: MERGE, CDC, time travel, schema enforcement




- Data Engineering & Development
- Strong Python and SQL programming skills
- Experience with ETL/ELT at scale: denormalization, surrogate keys, directory tables, curated data models
- Experience integrating complex data sources: Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, APIs
- Strong data modelling skills: relational, dimensional, and lakehouse-oriented

DevOps & Automation

- CI/CD pipelines for data engineering (Azure DevOps / GitHub Actions)
- Infrastructure as Code (Terraform or ARM)
- Containerization (Docker)
- Experience with automated testing frameworks for data pipelines (unit testing, reconciliation-based validation)

Nice to Have

- Experience with Unity Catalog for data governance and lineage
- Familiarity with Azure Purview for data cataloguing and governance
- Exposure to real-time and streaming pipelines (Event Hub / Kafka / Kinesis)
- Experience with GenAI or ML platform integration (MLOps, feature engineering pipelines)
- Familiarity with monitoring and observability tools (e.g., Dynatrace)
- Exposure to BI tools (Power BI, Tableau)

Experience & Profile

- 7+ years of experience in Data Engineering, with significant platform migration or re-engineering experience
- Proven track record auditing and taking ownership of existing, complex enterprise data platforms — not just building from scratch
- Deep knowledge of enterprise data governance patterns: audit trails, reconciliation, data quality controls, SCD versioning
- Strong analytical mindset: ability to read existing implementations, identify intent versus defect, and make sound re-engineering decisions
- Comfortable operating across both hands-on engineering and technical architecture
- Strong communication skills — able to engage business, governance, and engineering stakeholders with clarity
- Experience working in regulated or enterprise-scale environments (financial services a plus)

📌 Senior Data Engineer (Maharashtra)
🏢 Allianz Services
📍 Maharashtra