Job Summary

We are looking for an experienced Databricks & Apache Airflow Expert to design, develop, implement, and maintain scalable data engineering solutions. The ideal candidate should have robust hands-on experience with Databricks, Apache Spark, Python, SQL, and Apache Airflow, along with expertise in building and orchestrating complex ETL/ELT data pipelines.

The candidate will be responsible for developing robust data pipelines, optimizing data processing workflows, implementing data lakehouse architecture, and ensuring reliable data integration across multiple data sources.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and manage complex ETL/ELT workflows using Apache Airflow.
- Develop and orchestrate DAGs (Directed Acyclic Graphs) in Airflow.
- Implement data ingestion, transformation, and processing using PySpark and Python.




- Work extensively with Databricks Lakehouse architecture, Delta Lake, and Delta Live Tables (Lakeflow Declarative Pipelines).
- Develop and optimize SQL queries for large-scale data processing.
- Implement incremental data loading, CDC (Change Data Capture), and data transformation frameworks.
- Manage workflow scheduling, dependencies, retries, monitoring, and error handling in Airflow.
- Optimize Spark jobs and Databricks clusters for performance and cost efficiency.
- Implement data quality checks, validation, and monitoring mechanisms.
- Integrate data from various sources such as APIs, databases, cloud storage, and enterprise applications.
- Collaborate with data architects, analysts, and business stakeholders.
- Follow best practices for data security, governance, and access management.

📌 Databricks Engineer (Airflow) (India)
🏢 DataE2E Technologies
📍 India