Work as per key focus areas

Key focus areas:

- Strong hands-on experience with Databricks, Unity Catalog and Delta Lake
- Deep expertise in Unity Catalog governance, access controls, lineage and auditing
- Strong experience with PySpark, Python, SQL and Delta Live Tables
- Hands-on experience with Databricks Workflows, Auto Loader and Structured Streaming
- Strong knowledge of Lakehouse architecture, medallion architecture and data modelling
- Experience with Terraform/Databricks Asset Bundles, CI/CD and Git
- Experience with Databricks on AWS/Azure/GCP and platform cost optimisation is preferred

:

We are looking for an experienced Databricks Engineer to design, implement, and evolve enterprise-scale data platforms using the Databricks Lakehouse ecosystem. The role involves building scalable data pipelines, implementing Unity Catalog governance, and optimising Delta Lake workloads for reliability and performance. The candidate will work closely with data engineering, data science, analytics, and platform teams while leading technical initiatives.

Strong expertise in Databricks, Unity Catalog, Delta Lake, PySpark, and cloud data platforms is required.

Responsibilities

- Lead the architecture, implementation, and evolution of enterprise Databricks Lakehouse platforms.
- Design and implement Unity Catalog governance, including metastore architecture, access controls, data lineage, external locations, auditing, and data security.
- Build and optimise production-grade Delta Lake tables using schema evolution, partitioning, Z-ordering, compaction, vacuuming, and liquid clustering.
- Develop scalable data pipelines using PySpark, Delta Live Tables, Auto Loader, Spark Structured Streaming, and Databricks Workflows.




- Establish data quality frameworks, SLA monitoring, observability, and pipeline reliability standards.
- Manage Databricks platform operations including cluster policies, autoscaling, instance pools, workspace configuration, upgrades, and cost optimisation.
- Implement Infrastructure as Code and CI/CD using Terraform or Databricks Asset Bundles, Git, and deployment pipelines.
- Lead architecture reviews, technical decisions, and mentoring while collaborating with data scientists, analysts, and platform teams.

Requirements

- 8+ years of experience in Data Engineering, with at least 3 years of hands-on experience working with Databricks.
- Strong expertise in Unity Catalog, including metastore architecture, catalog/schema design, fine-grained access controls, external locations, lineage, and auditing.
- Expert knowledge of Delta Lake, including ACID transactions, transaction logs, schema evolution, optimisation, Delta Live Tables, and Delta Sharing.
- Strong hands-on experience with PySpark, Python, and SQL for large-scale data ingestion, transformation, aggregation, and streaming workloads.
- Experience with Databricks Workflows, Databricks SQL Warehouses, cluster management, autoscaling, policies, and platform cost optimisation.
- Experience implementing CI/CD and Infrastructure as Code using Terraform or Databricks Asset Bundles, Git, and cloud platforms.
- Robust understanding of Lakehouse architecture and data modelling, including medallion architecture, dimensional modelling, or data vault.
- Robust technical leadership and communication skills; Databricks certifications are preferred, with exposure to MLflow, Model Serving, Feature Store, Iceberg/Hudi, data mesh, or multi-cloud Databricks environments as an advantage.

📌 Lead Engineer / Associate Architect - Databricks (Bengaluru)
🏢 QBurst
📍 Bengaluru