- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
- Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
- Build and maintain Delta Lake tables following Bronze, Silver, and Gold/Medallion architecture patterns.
- Implement Delta Lake capabilities such as:
- MERGE operations
- Schema enforcement and schema evolution
- Time Travel
- OPTIMIZE and VACUUM
- Incremental data processing

- Develop and manage Databricks Workflows/Jobs for pipeline orchestration and scheduling.
- Troubleshoot pipeline failures, performance issues, and data quality problems.
- Optimize Spark workloads by understanding partitioning, joins, caching, shuffle operations, and query execution.
- Develop reusable data engineering utilities and frameworks using PySpark and basic Python.
- Write complex SQL queries for data transformation, reconciliation, validation, and analysis.




- Integrate Databricks pipelines with external systems using REST APIs.
- Handle inbound and outbound file integrations using SFTP.
- Work with common file formats such as CSV, JSON, Parquet, and Delta.
- Work with AWS services used within the data engineering ecosystem, particularly Amazon S3, and have basic awareness of services such as IAM, Secrets Manager, CloudWatch, and Lambda.
- Implement proper exception handling, logging, auditing, monitoring, and restart/recovery mechanisms.
- Participate in code reviews and follow coding and data engineering best practices.
- Collaborate with architects, business analysts, source-system teams, QA teams, and other engineering teams.
- Support production deployments and troubleshoot production data pipeline issues.

ETL

📌 SENIOR SOFTWARE ENGINEER - ETL (Bengaluru)
🏢 Happiest Minds Technologies
📍 Bengaluru