Roles & Responsibilities
- Develop Databricks notebooks, jobs, and workflows to replicate and enhance DB2/Guidewire-based pipelines and transformations.
- Implement Delta Lake tables and patterns (bronze/silver/gold, ACID, time travel, schema evolution) for migrated data.
- Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake as required.
- Optimize Databricks clusters, jobs, and queries for performance and cost.
- Implement incremental loads, CDC patterns, and batch schedules for large datasets.
- Collaborate with Snowflake and dbt teams to ensure consistent data models and data contracts.
- Participate in data validation and reconciliation between DB2 400 / Guiderwire and Databricks outputs.
- Follow coding standards, version control, and CI/CD practices using Git/Azure DevOps.
- Provide defect fixes and support during SIT/UAT and post go-live stabilization.

Experience :
Legacy Demystification & Ingestion (DB2/400 & Guidewire)





8+ years of experience in databricks with understanding complex legacy data models and getting that data into the cloud:
- Extracting DB2/AS400: Experience with Change Data Capture (CDC) or scheduled batch extractions from DB2 into cloud storage. Involves working through JDBC connections, mapping table dependencies, and re-platforming legacy SQL to distributed computing standards.
- Handling Guidewire Data: Integrating with Guidewire Cloud Data Access (CDA) or InsuranceSuite to replicate complex P&C; (Property & Casualty) insurance schemas. Senior engineers parse these highly normalized operational databases and transform them into analytical-friendly schemas in the cloud.
- Architecture & Pipeline Development

The core of the experience involves transitioning these legacy, row-based stores into a scalable Medallion Architecture (Bronze, Silver, Gold layers):
- Delta Lake Optimization: Using Databricks and Apache Spark to build ETL/ELT data pipelines with ACID transactions. Senior engineers handle schema evo

📌 Data Engineer Sr (Bangalore Urban)
🏢 NTT DATA North America
📍 Bangalore Urban