Role: Databricks Engineer

Experience: 9-12 Years

Location: ALL EXL Locations

Work Mode: Hybrid

Key Role and Responsibilities:

 Design, build, and maintain Databricks workspaces, clusters, and compute pools
across development, testing, and production environments.

 Configure and manage Unity Catalog for data governance, fine-grained access
control, permissions, metadata management, and data lineage.

 Optimize Databricks cluster configurations, including instance types,
auto-scaling, spot/preemptible nodes, and compute pools to improve performance
and reduce costs.

 Implement workspace best practices, including folder structures, access
controls, secret management using Databricks Secrets, Azure Key Vault, or AWS
Secrets Manager.

 Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job
orchestration with dependency management.

 Design and implement Delta Lake tables using partitioning, Z-Ordering,
OPTIMIZE, VACUUM, and file compaction techniques.

 Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers)




for scalable and governed data lakehouse solutions.

 Develop Delta Live Tables (DLT) pipelines with built-in data quality
expectations for reliable ETL/ELT processing.

 Manage schema evolution, table versioning, Time Travel, and Change Data Feed
(CDF) to support incremental data processing.

 Design and implement lakehouse architectures integrating Delta Lake with cloud
storage and external systems such as Azure Data Lake Storage (ADLS), Kafka,
Event Hubs, and Kinesis.

 Develop scalable batch and real-time data pipelines using PySpark, Spark SQL,
Structured Streaming, and Delta Lake.

 Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other
streaming platforms into Delta tables.
 Optimize PySpark applications using broadcast joins, Adaptive Query Execution
(AQE), agile partition pruning, caching, and Photon Engine.

 Develop reusable transformation frameworks, utility libraries, an

📌 Senior Manager (Pune)
🏢 EXL
📍 Pune