Job Summary

We are looking for an experienced Data Engineer with strong expertise in Apache Spark, Scala, Python, Databricks, Apache Airflow, Microsoft Fabric, SQL, and Azure Data Lake Storage (ADLS).

The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data engineering solutions across cloud and big-data platforms.

Key Responsibilities

- Design, develop, and maintain scalable ETL/ELT data pipelines using Apache Spark, Python, and Scala
- Develop and optimize Spark-based batch and distributed data processing applications
- Build and manage data pipelines and workflows using Apache Airflow
- Develop data engineering solutions using Azure Databricks, including notebooks, jobs, workflows, and Delta Lake
- Work with Microsoft Fabric for data ingestion, transformation, orchestration, and analytics workloads
- Design and manage data storage solutions using Azure Data Lake Storage (ADLS/ADLS Gen2)
- Write complex and optimized SQL queries for data transformation, validation, and analysis
- Implement data quality, validation, monitoring, and error-handling mechanisms
- Optimize Spark jobs, SQL queries, and data pipelines for performance and scalability
- Collaborate with architects, analysts, developers, and business stakeholders to understand data requirements
- Implement CI/CD and follow development, testing, deployment, and production-support best practices




- Troubleshoot pipeline failures and resolve data-processing and performance issues

Required Skills

- Strong hands-on experience with Apache Spark
- Robust programming experience in Scala and/or Python
- Hands-on experience with Apache Airflow and workflow orchestration
- Strong experience with Azure Databricks
- Hands-on experience with Microsoft Fabric
- Strong SQL development and query optimization skills
- Experience with Azure Data Lake Storage (ADLS/ADLS Gen2)
- Good understanding of ETL/ELT concepts and data engineering principles
- Experience working with large-scale distributed data processing environments
- Good understanding of data pipeline architecture, data quality, and performance optimization

Good to Have

- Experience with Delta Lake and Medallion Architecture
- Experience with Microsoft Fabric Data Factory, Lakehouse, and Warehouse
- Azure cloud services and data engineering ecosystem
- CI/CD using Azure DevOps or GitHub
- Experience with Kafka or other streaming technologies
- Knowledge of data governance and security practices
- Experience5 to 8 years of overall experience in Data Engineering / Big Data Engineering, with strong hands-on experience in the required technologies

Education

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Location - Pan India-Bengaluru,Hyderabad,Delhi / NCR,Chennai,Pune,Kolkata,Ahmedabad,Mumbai

📌 Data Engineer (Bengaluru)
🏢 Zorba consulting
📍 Bengaluru