Key Responsibilities
Databricks Platform Engineering

Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.

Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.

Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.

Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.

Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.

Data Pipeline & Integration

ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.

Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs,



and other Azure-native services.

Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.

Software Engineering & DevOps

Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.

CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.

Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.

Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, workplace-consistent deployments.

Data Governance & Quality

Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expect

📌 Senior Databricks Engineer (Bengaluru)
🏢 Chubb Fire & Security
📍 Bengaluru