We are looking for a skilled Data Engineer with strong experience in Python, SQL, Databricks, and Delta Lake. The candidate will be responsible for developing, implementing, and optimizing scalable data ingestion and transformation pipelines using Databricks and modern data engineering practices.The ideal candidate should have hands-on experience in data processing, ETL/ELT development, SQL query optimization, and working with large datasets.Key Responsibilities

- Design, develop, and maintain scalable data ingestion and transformation pipelines using Python and Databricks.
- Implement ETL/ELT processes to ingest data from multiple sources into the Databricks platform.
- Develop PySpark and Python scripts for data extraction, transformation, and loading.
- Work with Delta Lake tables for reliable data storage and processing.
- Implement data processing using Bronze, Silver, and Gold Medallion architecture.
- Develop and optimize complex SQL queries, stored procedures, joins, and window functions.
- Perform data cleansing, validation, deduplication, and reconciliation.
- Implement incremental data loads and handle historical data processing.
- Troubleshoot data pipeline failures and improve job performance.
- Collaborate with cross-functional teams to deliver high-quality data solutions.

Technical Skills & Requirements1. Python

- Robust knowledge of Python programming.
- Experience with functions, modules, classes, exception handling, and file processing.
- Knowledge of Python data structures and collections.
- Ability to write reusable, modular, and optimized Python code.
- Experience working with JSON, CSV, and other file formats.

2.

Data Engineering

- Strong understanding of ETL/ELT concepts and data pipeline development.
- Experience with batch data processing.
- Understanding of incremental and full data loads.
- Knowledge of data validation, data cleansing, and data quality checks.
- Understanding of structured and semi-structured data.
- Basic understanding of data warehouse concepts.

3. Databricks & Data Ingestion





- Hands-on experience with Databricks platform.
- Experience developing data ingestion pipelines using PySpark and Databricks.
- Knowledge of reading and writing data in CSV, JSON, Parquet, and other formats.
- Understanding of Databricks notebooks, jobs, and workflows.
- Experience ingesting data from databases, files, and external sources.
- Understanding of incremental ingestion and schema evolution.
- Knowledge of Bronze, Silver, and Gold layers.

4. SQL

- Strong knowledge of SQL queries and relational databases.
- Hands-on experience with different types of joins:
- INNER JOIN
- LEFT JOIN
- RIGHT JOIN
- FULL OUTER JOIN

- Experience with subqueries and Common Table Expressions (CTEs).
- Strong understanding of window functions such as ROW_NUMBER, RANK, DENSE_RANK, and LEAD/LAG.
- Experience developing and debugging stored procedures.
- Knowledge of GROUP BY, HAVING, aggregate functions, and CASE statements.
- Basic understanding of data modeling concepts, including fact and dimension tables.
- Ability to analyze and optimize SQL queries.

5.

Delta

Lake

- Understanding of Delta Lake architecture and capabilities.
- Hands-on experience creating and managing Delta tables.
- Knowledge of Delta Lake operations such as INSERT, UPDATE, DELETE, and MERGE.
- Understanding of ACID transactions.
- Knowledge of schema enforcement and schema evolution.
- Understanding of Delta Lake time travel and history.
- Understanding of OPTIMIZE and VACUUM operations.
- Knowledge of implementing incremental loads and handling duplicate records.

Good to Have

- Experience with Azure Data Factory or AWS data services.
- Knowledge of Git and CI/CD processes.
- Understanding of data warehousing and dimensional modeling.
- Experience with Databricks Workflows and job scheduling.
- Knowledge of Spark performance optimization.

Candidate Expectations

- Strong analytical and problem-solving skills.
- Ability to develop clean, reusable, and maintainable code.
- Good communication and collaboration skills.
- Ability to troubleshoot and resolve data engineering issues independently.

📌 MODULE LEAD - Databricks (Bengaluru)
🏢 Happiest Minds
📍 Bengaluru