Key Responsibilities:
Databricks Platform Engineering
•
Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
•
Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
•
Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.
•
Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
•
Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
Data Pipeline & Integration
•
ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
•
Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse,



Event Hubs, and other Azure-native services.
•
Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
Software Engineering & DevOps
•
Code Quality: Write clean, productive, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
•
CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
•
Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
•
Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
Data Governance & Quality
•
Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using G

📌 Senior Databricks Engineer (Bengaluru)
🏢 Chubb Fire & Security Group
📍 Bengaluru