Job Description: Senior Databricks Data Engineer

Role Overview

We are looking for an experienced Senior Databricks Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake. The candidate should have robust hands-on experience in data transformation, pipeline orchestration, performance optimization, data quality, and production support within an AWS-based data engineering environment.

The role also requires practical experience using Cursor, GitHub Copilot, Windsurf, or similar agentic AI-enabled development tools to accelerate development, testing, debugging, documentation, and code optimization.

Key Responsibilities

Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
Build and maintain Delta Lake tables following Bronze, Silver, and Gold layers of the Medallion Architecture.
Implement Delta Lake capabilities, including:
MERGE and upsert operations




Schema enforcement and schema evolution
Time Travel and version management
OPTIMIZE and VACUUM
Incremental and change-based data processing
Develop and manage Databricks Workflows and Jobs for pipeline orchestration, scheduling, dependency management, retries, and alerting.
Design efficient incremental-load and restart/recovery mechanisms to ensure reliable pipeline execution.
Troubleshoot pipeline failures, performance bottlenecks, data-quality issues, and production incidents.
Optimize Spark workloads through effective use of partitioning, joins, caching, broadcast strategies, shuffle management, file sizing, and query execution plans.
Develop reusable data engineering utilities, libraries, and frameworks using PySpark and Python.
Write complex SQL queries for data transformation, reconciliation, validation, analysis, and reporting.
Integrate Databricks pipelines with external applications and systems using REST API

📌 DATA ENGINEER (Bengaluru)
🏢 Happiest Minds Technologies
📍 Bengaluru