Role Overview

We are looking for experienced Azure Databricks Data Engineers with strong hands-on expertise in Apache Spark/PySpark, Python, SQL, and Azure Data Factory (ADF). The candidate will be responsible for designing, developing, and optimizing scalable data engineering solutions on the Azure cloud platform.

Mandatory Skills

- Strong hands-on experience with Azure Databricks
- Strong knowledge of Apache Spark and/or PySpark
- Proficiency in Python
- Strong SQL development and query optimization skills
- Hands-on experience with Azure Data Factory (ADF)
- Experience developing and maintaining scalable ETL/ELT data pipelines
- Good understanding of Azure data engineering concepts and cloud-based data platforms

Key Responsibilities

- Design, develop, and maintain data pipelines using Azure Databricks and ADF
- Develop efficient data processing solutions using PySpark/Spark and Python
- Write complex SQL queries for data transformation, validation, and analysis
- Build scalable ETL/ELT workflows for large datasets
- Optimize Spark jobs, Databricks notebooks, and data pipelines for performance
- Integrate data from multiple sources into Azure-based data platforms
- Implement data quality, validation,



error handling, and monitoring mechanisms
- Troubleshoot production issues and provide timely resolutions
- Collaborate with data architects, analysts, developers, and business stakeholders
- Follow best practices for code quality, security, performance, and maintainability

Experience Requirements

For 5+ Years

- 5+ years of overall experience in data engineering
- Strong hands-on experience in Azure Databricks, PySpark/Spark, Python, SQL, and ADF
- Experience working on enterprise-scale data pipelines

For 9+ Years

- 9+ years of overall experience in data engineering
- Strong expertise in Azure Databricks and up-to-date Azure data engineering
- Proven experience designing and optimizing large-scale data pipelines
- Ability to lead technical discussions and mentor junior engineers

Preferred Skills

- Azure Data Lake Storage (ADLS)
- Delta Lake
- Databricks Workflows
- Azure DevOps / CI-CD
- Git
- Data warehousing concepts
- Experience with Agile/Scrum methodologies

Skills:- Azure Databricks, Azure data factory, Python, PySpark, Apache Spark and SQL

📌 Azure Data Engineer (Bengaluru)
🏢 VY SYSTEMS PRIVATE
📍 Bengaluru