Location : Noida

Work Mode : Work From Office

Experience : 9-12 Years

Role Overview :

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Python and SQL to design, develop and optimize scalable enterprise data engineering solutions.

The ideal candidate should have extensive experience in building robust ETL/ELT pipelines, processing large-scale datasets, implementing contemporary Lakehouse architectures and optimizing data platforms for performance, scalability and cost efficiency.

The candidate will also drive Databricks serverless migration initiatives, CI/CD implementation and engineering best practices.

Key Responsibilities :

- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark, Python and SQL.
- Build and optimize ETL/ELT workflows for structured and semi-structured data.
- Develop and manage data pipelines using Apache Spark, Spark SQL and Delta Lake.
- Lead migration of existing Databricks workloads to serverless compute architecture.
- Implement and maintain CI/CD pipelines using GitHub Actions or equivalent tools.
- Manage code repositories, monitor pull requests, conduct code reviews and ensure deployment quality.
- Optimize Spark jobs, data processing workflows and Databricks workloads for performance and cost efficiency.
- Implement data governance, security, access control and compliance best practices.
- Develop modular, reusable and maintainable code; refactor existing pipelines where required.
- Work closely with cross-functional teams to understand business requirements and deliver reliable data solutions.
- Support automated testing, data validation, monitoring and production troubleshooting.

Mandatory Technical Skills :

- Azure Databricks :



Minimum 2+ years of hands-on experience.
- Data Engineering : 8+ years of relevant experience in Data Engineering, ETL/ELT and large-scale data processing.
- Programming : Strong hands-on expertise in Python and PySpark.
- Apache Spark : Strong understanding of Spark architecture, Spark SQL and performance optimization.
- SQL : Advanced SQL skills, including complex queries and data transformations.
- ETL/ELT : Experience designing and implementing enterprise-grade data pipelines.
- Data Architecture : Strong understanding of Delta Lake, Lakehouse architecture and data modelling.
- Orchestration : Experience with Apache Airflow and/or equivalent orchestration tools.
- CI/CD : Hands-on experience with GitHub Actions, Azure DevOps or similar CI/CD platforms.

Preferred / Good-to-Have Skills :

- Delta Live Tables (DLT).
- Unity Catalog and data governance.
- Delta Sharing.
- Databricks SQL Warehouse, including serverless SQL Warehouse.
- Serverless compute architecture and migration.
- dbt or similar data transformation frameworks.
- Cloud security, access management and compliance.
- Experience with financial data, pricing logic or complex business calculations.
- Exposure to AI/ML-based data solutions.
- Experience leading technical teams, mentoring engineers and conducting code reviews.

Ideal Candidate Profile :

- Strong hands-on engineering background rather than only project coordination or team management.
- Proven experience developing production-grade Databricks and PySpark solutions.
- Experience handling large-scale data migration and transformation projects.
- Strong problem-solving, analytical and communication skills.
- Ability to independently drive technical initiatives and collaborate with stakeholders.

📌 Senior Data Engineer - Azure Databricks (India)
🏢 wowJobs
📍 India