– Azure Databricks Data Engineer

Experience Required

Total Experience: 8–12 Years

Relevant Experience: 6+ Years

Role: Azure Data Engineer / Azure Databricks Engineer

Mandatory Skills (Top Screening Skills)

Azure Databricks

PySpark / Python

Data Modelling

SQL

Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF)

Job Overview

We are looking for an experienced Azure Data Engineer with strong expertise in Azure Data Engineering, Databricks, PySpark, SQL, and Data Modelling. The ideal candidate should have hands-on experience in designing and implementing scalable ELT pipelines, data integration solutions, and data processing workflows using the Azure ecosystem.

The candidate should be capable of owning end-to-end data engineering solutions, ensuring data quality, performance optimization, and efficient data pipeline execution.

Key Responsibilities

Design, develop, and maintain scalable data engineering solutions using Azure Databricks and Azure services.

Build and optimize ELT pipelines using Azure Data Factory and Databricks.

Develop complex data transformation and processing workflows using PySpark and Python.

Design and implement robust data models to support business and analytical requirements.

Work extensively with Azure Data Lake Storage (ADLS) for data storage and management.

Develop, optimize, and troubleshoot complex SQL queries and database operations.

Handle data integration, ingestion, manipulation, and processing across multiple data sources.

Implement data pipeline orchestration and workflow automation.

Ensure data quality, consistency, reliability, and performance across data platforms.

Implement CI/CD pipelines using GitHub Actions and Azure DevOps.





Work with PyTest for unit testing and SonarQube for code quality and analysis.

Implement and manage Databricks Asset Bundles for deployment and release management.

Collaborate with cross-functional teams and stakeholders to deliver scalable data solutions.

Follow engineering best practices, coding standards, and documentation guidelines.

Technical Skills Required

Mandatory / Core Skills

Azure Cloud – Mastery

ELT – Mastery

Data Modelling – Mastery

Data Integration and Ingestion – Robust

Data Manipulation and Processing – Strong

Azure Databricks

PySpark / Python

SQL

Azure Data Lake Storage (ADLS)

Azure Data Factory (ADF)

Additional Technical Skills

GitHub and GitHub Actions

Azure DevOps

Azure SQL Database

PyTest

Azure Stream Analytics

AWS Glue

Apache Airflow

Amazon Kinesis

Amazon Redshift

SonarQube

Databricks Asset Bundles

Good-to-Have Skills

Generative AI concepts and implementation.

Databricks Genie.

Databricks Mosaic AI.

GitHub Copilot.

Power BI.

CI/CD implementation and automation.

Experience with external technical ecosystems.

Experience in running and managing Scrum teams.

MKDocs documentation and technical documentation practices.

Preferred Candidate Profile

Strong problem-solving and analytical skills.

Ability to independently design and deliver end-to-end data engineering solutions.

Good communication and stakeholder management skills.

Experience working in Agile/Scrum environments.

Ability to manage technical discussions and collaborate with distributed teams.

Strong understanding of cloud-based data engineering architecture and best practices.

Key Focus Areas

Azure Data Engineering

Azure Databricks and PySpark

ELT Pipeline Development

Data Modelling

Data Integration and Processing

CI/CD and DevOps

Cloud Data Architecture

📌 Azure Data Engineer / Azure Databricks Engineer (Bengaluru)
🏢 Consultbae India
📍 Bengaluru