Azure Databricks Developer

Location: India

Work Mode: 100% Remote

Shift: EST Time Zone

Experience: 9–10+ Years

Engagement: Contract

Job Overview

We are seeking an experienced Azure Databricks Developer with strong hands-on expertise in Azure Data Engineering, Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory (ADF), and ADLS Gen2 .

The successful candidate will be responsible for designing, developing, optimizing, and supporting scalable enterprise-grade data engineering solutions using modern Lakehouse Architecture . The role requires strong experience in ETL/ELT development, incremental processing, CDC, Spark performance optimization, and Azure cloud services.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark .
- Build and orchestrate robust ETL/ELT workflows using Azure Data Factory (ADF) .
- Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL .
- Implement Medallion Architecture (Bronze, Silver, Gold) and modern Lakehouse design patterns.
- Develop incremental data processing, CDC, MERGE operations, and SCD Type 1/Type 2 implementations.
- Optimize Spark workloads using partitioning, caching, file-size optimization, and other performance-tuning techniques.
- Configure and manage Databricks Jobs/Workflows, clusters, and workspace environments .
- Implement data quality checks, validation, reconciliation, error handling, monitoring, and alerting processes.
- Work with Unity Catalog for data governance, lineage, RBAC, and secure data access.
- Integrate Databricks with ADLS, Azure SQL, APIs, JDBC sources, and external systems .




- Develop and maintain CI/CD pipelines using Azure DevOps, Git, and YAML .
- Implement Infrastructure-as-Code using Terraform, ARM, or Bicep .
- Troubleshoot production data pipelines, perform root-cause analysis, and ensure SLA compliance.
- Monitor and optimize workloads for performance, scalability, reliability, and cost efficiency .
- Collaborate with architects, BI teams, data analysts, developers, and business stakeholders in an Agile/Scrum environment .

Required Skills & Qualifications
- 9–10+ years of experience in Data Engineering / Azure Cloud Data Development.
- Strong hands-on experience with:
- Azure Databricks
- PySpark
- Spark SQL
- Delta Lake
- Azure Data Factory (ADF)
- ADLS Gen2
- SQL
- Strong understanding of Lakehouse Architecture and distributed data processing .
- Proven experience developing enterprise-scale ETL/ELT pipelines .
- Hands-on experience with Medallion Architecture, CDC, MERGE, and SCD Type 1/Type 2 .
- Strong knowledge of Apache Spark performance tuning and optimization .
- Experience with Azure DevOps, Git, and YAML-based CI/CD pipelines .
- Knowledge of Terraform, ARM, or Bicep for Infrastructure-as-Code.
- Solid SQL development and query performance-tuning capabilities.
- Experience working in Agile/Scrum environments .




- Strong troubleshooting and production-support experience.

Preferred Skills
- Experience with Databricks platform administration , including clusters, jobs, workflows, and workspace configuration.
- Hands-on experience with Unity Catalog, RBAC, governance, and data lineage .
- Experience integrating Databricks with APIs, JDBC sources, Azure SQL, and external systems .
- Knowledge of monitoring and observability tools such as Datadog or Grafana .
- Experience with Azure and Databricks cost optimization .
- Experience supporting production-grade data platforms and resolving critical data pipeline issues.

Candidate Profile The ideal candidate should be:
- Strongly hands-on with Azure Databricks and PySpark .
- Experienced in designing and supporting enterprise-scale data pipelines .
- Strong in Spark optimization, SQL, Delta Lake, and Lakehouse architecture .
- Comfortable working in an EST time-zone shift .
- Capable of independently troubleshooting production issues and meeting defined SLAs.
- Comfortable collaborating with cross-functional technical and business teams.
- Experienced in working in fast-paced Agile/Scrum environments .

Technology Stack Azure Databricks | PySpark | Spark SQL | Delta Lake | Azure Data Factory | ADLS Gen2 | Azure SQL | SQL | Unity Catalog | Azure DevOps | Git | YAML | Terraform | ARM | Bicep | Lakehouse Architecture | CDC | SCD Type 1/2

Engagement Details

ParameterDetails Role :- Azure Databricks Developer

Experience 9–10+ Years Location India Work Mode 100% Remote Shift EST Engagement Contract Joining Preference Immediate Joiners Preferred

📌 Lead Azure Databricks Developer ( 9+ Years) (India)
🏢 MindBrain
📍 India