Job Description

Job Description – Senior Data Engineer / Platform Re-Engineering Lead (Azure Synapse & Databricks Migration)

n

Role Overview We are looking for a highly experienced Senior Data Engineer to lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics. The role requires deep technical seniority to audit, understand, and validate a complex end-to-end data architecture spanning source ingestion through to consumption — and to drive a future migration of validated workloads to Databricks. This is not a greenfield role: it demands the ability to reverse-engineer existing implementations, assess their correctness, and own the technical migration strategy.

n

n

Key Responsibilities

n

n
- Lead the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
n
- Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
n
- Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
n
- Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
n
- Design and implement target-state architectures aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
n
- Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
n
- Define and execute migration strategies for existing workloads to modern data platforms, preserving existing governance and control framework semantics
n
- Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
n
- Collaborate with business, data governance,



and architecture stakeholders to validate embedded business rules and data quality requirements
n
- Provide technical leadership across re-engineering and migration workstreams, contributing to decommission planning for legacy components
n

n

n

n

n

Core Technical Skills

n

n
- Azure Synapse & Data Platform Mandatory hands-on expertise with:
n
- Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool)
n
- Azure Data Lake Storage Gen2 (ADLS Gen2)
n
- Delta Lake on Azure (Synapse Lakehouse patterns)
n
- Oracle Golden Gate Replication for real-time source integration
n
- Azure Analysis Services and Power BI consumption layer patterns
n
- Deep understanding of medallion architecture: Raw / Harmonized / Conformed / Consumption layers
n
- Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard delete, and retroactive change processing
n
- Experience with Synapse SQL Pool — stored procedures, control tables, and data quality validation patterns
n
- Experience with audit, balance, and control frameworks — parameterized, modular pipeline governance at enterprise scale
n
- Familiarity with config-driven and automation-first pipeline patterns (YAML, PySpark, SQL-driven generation from mapping documents)
n

n

n

Databricks & Lakehouse

n

n
- Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred)
n
- Strong Apache Spark skills (PySpark / Spark SQL)
n
- Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse
n
- Ability to re-implement governance and control frameworks natively in Databricks (audit logging, reconciliation, DQ checks)
n
- Experience with Delta Lake features: MERGE, CDC, time travel, schema enforcement
n




- Data Engineering & Development
n
- Strong Python and SQL programming skills
n
- Experience with ETL/ELT at scale: denormalization, surrogate keys, directory tables, curated data models
n
- Experience integrating complex data sources: Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, APIs
n
- Strong data modelling skills: relational, dimensional, and lakehouse-oriented
n

n

n

DevOps & Automation

n

n
- CI/CD pipelines for data engineering (Azure DevOps / GitHub Actions)
n
- Infrastructure as Code (Terraform or ARM)
n
- Containerization (Docker)
n
- Experience with automated testing frameworks for data pipelines (unit testing, reconciliation-based validation)
n

n

n

n

Nice to Have

n

n
- Experience with Unity Catalog for data governance and lineage
n
- Familiarity with Azure Purview for data cataloguing and governance
n
- Exposure to real-time and streaming pipelines (Event Hub / Kafka / Kinesis)
n
- Experience with GenAI or ML platform integration (MLOps, feature engineering pipelines)
n
- Familiarity with monitoring and observability tools (e.g., Dynatrace)
n
- Exposure to BI tools (Power BI, Tableau)
n

n

n

Experience & Profile

n

n
- 7+ years of experience in Data Engineering, with significant platform migration or re-engineering experience
n
- Proven track record auditing and taking ownership of existing, complex enterprise data platforms — not just building from scratch
n
- Deep knowledge of enterprise data governance patterns: audit trails, reconciliation, data quality controls, SCD versioning
n
- Strong analytical mindset: ability to read existing implementations, identify intent versus defect, and make sound re-engineering decisions
n
- Comfortable operating across both hands-on engineering and technical architecture
n
- Solid communication skills — able to engage business, governance, and engineering stakeholders with clarity
n
- Experience working in regulated or enterprise-scale environments (financial services a plus)
n

n

📌 Senior Data Engineer (Maharashtra)
🏢 Allianz Services
📍 Maharashtra