Job Title: Databricks Engineer
Department: Client Delivery / Data & AI Engineering
Experience Required: 8-15 Years (overall) | 5+ Years Databricks hands-on
Employment Type: Full-Time
Location: Hybrid (as per business requirement)
Reporting To: Delivery Manager / Technical Project Manager
Notice Period- Immediate to 15 Days
Key Responsibilities
Databricks Platform Engineering
Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.
Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.
Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance.
Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).
Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.
Delta Lake & Lakehouse Architecture
Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).
Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.
Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.
Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.
Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).
Data Pipeline Development (PySpark / SQL)
Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake.
Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.
Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and agile partition pruning.
Create reusable transformat
📌 Databricks Engineer (Pune)
🏢 EXL
📍 Pune