Role & responsibilities
Design, build, and implement Databricks-based data ingestion and transformation pipelines using Delta Lake, Delta Live Tables (DLT), Auto Loader, and Unity Catalog.
Migrate legacy SQL Server ETL workloads and file-based data pipelines to a scalable Databricks Lakehouse architecture.
Develop incremental data ingestion solutions using Delta Change Data Feed (CDF) to replace traditional file-based change detection mechanisms.
Build and optimize bronze, silver, and gold data layers following Lakehouse best practices.
Configure and optimize JDBC connectivity between Databricks and SQL Server for large-scale enterprise datasets.
Design scalable partitioning, clustering (Z-Ordering), and performance optimization strategies for high-volume Delta tables.
Develop Delta MERGE processes to support inserts, updates,
and deletes while ensuring data consistency.
Replace legacy Windows Task Scheduler jobs with automated Databricks workflows and Delta Live Tables pipelines.
Perform data validation, reconciliation, row-count verification, and parallel-run testing to ensure migration accuracy.
Enable secure enterprise data sharing using Unity Catalog, Databricks SQL Warehouse, and JDBC connectivity.
Collaborate with business stakeholders, application owners, architects, and platform teams to deliver end-to-end migration solutions.
Document architecture, migration strategies, operational runbooks, and production support procedures.