Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
Build and maintain Delta Lake tables following Bronze, Silver, and Gold/Medallion architecture patterns.
Implement Delta Lake capabilities such as:
MERGE operations
Schema enforcement and schema evolution
Time Travel
OPTIMIZE and VACUUM
Incremental data processing
Develop and manage Databricks Workflows/Jobs for pipeline orchestration and scheduling.
Troubleshoot pipeline failures, performance issues, and data quality problems.
Optimize Spark workloads by understanding partitioning, joins, caching, shuffle operations, and query execution.
Develop reusable data engineering utilities and frameworks using PySpark and basic Python.
Write complex SQL queries for data transformation, reconciliation, validation, and analysis.




Integrate Databricks pipelines with external systems using REST APIs.
Handle inbound and outbound file integrations using SFTP.
Work with common file formats such as CSV, JSON, Parquet, and Delta.
Work with AWS services used within the data engineering ecosystem, particularly Amazon S3, and have basic awareness of services such as IAM, Secrets Manager, CloudWatch, and Lambda.
Implement proper exception handling, logging, auditing, monitoring, and restart/recovery mechanisms.
Participate in code reviews and follow coding and data engineering best practices.
Collaborate with architects, business analysts, source-system teams, QA teams, and other engineering teams.
Support production deployments and troubleshoot production data pipeline issues.
ETL

📌 SENIOR SOFTWARE ENGINEER (Bengaluru)
🏢 Happiest Minds Technologies
📍 Bengaluru