Consultant/Senior Consultant | Databricks |Coimbatore | Engineering | Data Modernization & Migration

- Job requisition ID : 111044
- Location: Delhi
- Entity: Deloitte Touche Tohmatsu India LLP

Job Title: Senior Databricks Engineer

Experience: 6-9 Years

Location: Mumbai

Employment Type: Full-Time

Job Summary

We are seeking an experienced Databricks Developer with 6-9 years of expertise in designing, building, and supporting modern Data Lakehouse solutions. The ideal candidate will have robust hands-on experience in Databricks, PySpark, Delta Lake, cloud platforms, and large-scale data processing. The role involves building scalable data pipelines, optimizing distributed data workloads, and enabling advanced analytics and AI-driven initiatives.

Key Responsibilities

- Design, develop, and maintain scalable data engineering solutions using Databricks.

- Build and optimize batch and real-time data processing pipelines.

- Develop ETL/ELT frameworks using PySpark and Spark SQL.

- Implement Delta Lake architecture and data governance standards.

- Process large-scale structured, semi-structured, and streaming datasets.

- Optimize Spark jobs and cluster performance for maximum efficiency.

- Develop reusable frameworks, notebooks, workflows, and data products.

- Collaborate with business teams, data scientists, analysts, and architects to deliver enterprise-grade solutions.

- Implement monitoring, alerting, security, and compliance requirements.

- Participate in code reviews, deployment activities, and production support.

- Develop technical documentation and mentor junior engineers.

Required Technical Skills

Databricks & Big Data

- Azure Databricks

- Databricks Lakehouse Platform

- Delta Lake

- Spark SQL

- PySpark

- Databricks Workflows

- Unity Catalog

- Delta Live Tables (DLT)

- Structured Streaming

Data Engineering
- Data Lake Architecture

- Lakehouse Architecture





- ETL/ELT Development

- Data Modeling

- Data Integration

- Performance Tuning

- Data Quality Frameworks

Programming Languages
- Python

- PySpark

- SQL

- Scala

- Java

- Shell Scripting

Cloud Platforms
- Microsoft Azure

- Azure Data Factory (ADF)

- Azure Data Lake Storage (ADLS)

- AWS

- Amazon S3

- AWS EMR

- Google Cloud Platform (GCP)

Databases
- SQL Server

- Oracle

- PostgreSQL

- MySQL

- MongoDB

- Cassandra

Streaming & Messaging
- Kafka

- Event Hubs

- Spark Streaming

- Azure Stream Analytics

DevOps & Tools
- Git

- Jenkins

- Azure DevOps

- CI/CD Pipelines

- Terraform

- Unix/Linux

Required Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.
- 6-9 years of IT experience with at least 4 years of hands-on Databricks and Spark development experience.
- Strong expertise in distributed data processing and Lakehouse architecture.
- Experience building enterprise-scale data engineering solutions.
- Strong knowledge of PySpark, Spark SQL, and performance optimization.
- Experience with Agile development methodologies.
- Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications

- Experience with cloud-native analytics platforms on Azure, AWS, or GCP.
- Knowledge of Machine Learning workflows within Databricks.
- Experience implementing Delta Lake and Unity Catalog.
- Exposure to Data Governance and Data Security frameworks.
- Relevant Databricks, Azure, or AWS certifications.

Key Competencies

- Data Engineering Excellence

- Performance Optimization

- Solution Design

- Stakeholder Management

- Collaboration and Leadership

- Ownership and Accountability

Nice to Have
- Snowflake

- Azure Data Factory (ADF)

- Delta Live Tables (DLT)

- Unity Catalog

- MLflow

- Airflow

- Kafka

- Data Mesh Architecture

- Generative AI / AI Engineering exposure

- dbt

📌 Consultant/Senior Consultant | Databricks |Coimbatore | Engineering | Data Modernization & Migration (Delhi)
🏢 Deloitte
📍 Delhi