Senior Databricks Engineer

Exp-9-15 years

Work location- Bangalore

We are looking for a Senior Databricks Engineer to become a cornerstone of our Data & Analytics team. In this role, you will own the design and delivery of our Databricks-first data platform - building high-performance data pipelines, architecting scalable Lakehouse solutions, and championing engineering best practices across the team. You will work closely with business analysts, a data modeler and business stakeholders to turn raw data into reliable, production-grade analytics products.

Responsibilities

- Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
- Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
- Unity Catalog: Govern data assets - tables, volumes, and models - through Unity Catalog, enforcing fine-grained access control and lineage tracking.
- Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
- Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
- ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
- Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
- Streaming:



Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
- Code
- Write clean,
- and
- coding standards.
- CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
- Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
- Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
- Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Outstanding Expectations or DLT expectations.
- Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
- Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
- Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
- Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.




- Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.

Requirements

- 10+
- Azure + Databricks Certified (Mandatory)
- Databricks Certified Data Engineer Professional or Associate (mandatory).
- 10+ years of hands-on experience in Data Engineering roles.
- Deep expertise in Databricks - Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
- Advanced PySpark and Python skills for large-scale distributed data processing.
- Strong SQL skills: query authoring, performance tuning, and data modeling.
- Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
- Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
- Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
- Experience with Structured Streaming or other real-time data processing frameworks.
- Experience with Microsoft Purview for data cataloguing and governance.
- Familiarity with dbt (data build tool) for SQL-based transformation workflows.
- Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
- Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
- Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
- Background in data mesh or data product thinking.
- Databricks,
- PySpark, Python, SQL
- Azure
- DevOps
- Azure
- Terraform, Bicep, DABs

Skills: Databricks , PySpark , Python

📌 Senior Databricks Engineer (Bengaluru)
🏢 Value Vision
📍 Bengaluru