Role - Senior Databricks Engineer

Experience Required: 8+ Years

Location: Hyderabad

Employment Type: Full Time, Permanent

Job Overview

We are seeking a Senior Databricks Engineer (8+ Years) to lead end-to-end enterprise data platform solutions across Azure and AWS environments. You will architect high-performance Lakehouse platforms, build scalable ETL/ELT pipelines using PySpark and Delta Lake, drive client/stakeholder engagements, and mentor technical teams.

Key Responsibilities

- Technical Leadership & Delivery: Lead the end-to-end design, development, and deployment of enterprise data platforms using Databricks, PySpark, SQL, and Delta Lake.
- Stakeholder Management: Conduct technical discussions, architecture reviews, effort estimations, and requirement-gathering workshops directly with clients and business stakeholders.
- Architecture & Optimization: Design scalable Lakehouse/Medallion architectures; optimize Spark applications, Databricks Workflows, and Job Clusters to minimize infrastructure costs and maximize speed.
- Governance & Security: Implement Unity Catalog for access control, metadata management, and data lineage while upholding ISO 27001 / ISMS compliance standards.
- Cloud & Pipeline Engineering:



Build robust batch and real-time ETL pipelines integrating Databricks with Azure (ADF, Synapse) and/or AWS (S3, Glue, Lambda).
- DevOps & Best Practices: Drive automation via CI/CD pipelines (GitLab/GitHub), enforce coding standards, and mentor junior/mid-level data engineers.

Key Requirements

- 8+ years of total experience in Data Engineering & Big Data Analytics.
- 5+ years of hands-on expertise with Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
- Deep understanding of Spark internals, cluster tuning, partitioning strategies, and Medallion Architecture.
- Proven hands-on experience with Azure (ADF/Synapse) or AWS (S3/Glue/Lambda).
- Solid experience with Unity Catalog, CI/CD DevOps practices, and data governance frameworks.
- Robust client-facing, communication, and technical team leadership capabilities.
- Education: Bachelors or Master’s degree in Computer Science, IT, or a related field.

Key Skills - Databricks PySpark Apache Spark Delta Lake Python SQL Azure Data Factory (ADF) AWS Glue Unity Catalog Medallion Architecture Data Engineering Lakehouse Architecture CI/CD ISO 27001

📌 Senior Data Bricks Engineer (Hyderabad)
🏢 Databeat
📍 Hyderabad