Role & responsibilities

- Design, build, and implement Databricks-based data ingestion and transformation pipelines using Delta Lake, Delta Live Tables (DLT), Auto Loader, and Unity Catalog.

- Migrate legacy SQL Server ETL workloads and file-based data pipelines to a scalable Databricks Lakehouse architecture.

- Develop incremental data ingestion solutions using Delta Change Data Feed (CDF) to replace traditional file-based change detection mechanisms.

- Build and optimize bronze, silver, and gold data layers following Lakehouse best practices.

- Configure and optimize JDBC connectivity between Databricks and SQL Server for large-scale enterprise datasets.

- Design scalable partitioning, clustering (Z-Ordering), and performance optimization strategies for high-volume Delta tables.

- Develop Delta MERGE processes to support inserts, updates,



and deletes while ensuring data consistency.

- Replace legacy Windows Task Scheduler jobs with automated Databricks workflows and Delta Live Tables pipelines.

- Perform data validation, reconciliation, row-count verification, and parallel-run testing to ensure migration accuracy.

- Enable secure enterprise data sharing using Unity Catalog, Databricks SQL Warehouse, and JDBC connectivity.

- Collaborate with business stakeholders, application owners, architects, and platform teams to deliver end-to-end migration solutions.

- Document architecture, migration strategies, operational runbooks, and production support procedures.

📌 Lead/ Senior Data Engineer ( Actvie Databricks required) _ Exp: 7+ (Bengaluru)
🏢 Atyeti
📍 Bengaluru