This is a remote position.

Location: India

Work Mode: Remote

Employment Type: Full-time

Experience: 5 to 10 Years

About the Role

We are seeking an experienced Senior Data Engineer – Azure Databricks to design, develop, and optimize enterprise-scale data engineering solutions and up-to-date cloud-based data platforms. The ideal candidate brings deep, hands-on expertise in Azure Databricks, PySpark, Python, SQL, Azure Data Factory (ADF), and Azure Cloud, alongside strong experience building scalable data pipelines and implementing automated CI/CD via Azure Dev Ops.

In this role, you will work closely with data architects, engineers, analysts, and business stakeholders to build reliable, high-performance data platforms, optimize distributed Spark workloads, and mentor growing engineering talent.

Key Responsibilities

- Data Pipeline Engineering: Design, build, and maintain scalable end-to-end ETL/ELT pipelines and complex processing workflows using Azure Databricks, PySpark, Python, and SQL.

- Orchestration & Integration: Build and orchestrate data ingestion workflows using Azure Data Factory (ADF) to integrate data from diverse structured and unstructured enterprise sources into Azure Data Lake Storage (ADLS Gen2).

- Lakehouse Architecture & Modeling: Implement scalable Lakehouse and Medallion architectures, designing data models and data warehousing solutions to support downstream analytics, reporting, and business applications.

- Performance & Cost Optimization: Audit and optimize Databricks jobs, Spark workloads, SQL queries, memory management, and data pipelines for peak performance and cloud cost efficiency.

- CI/CD & Dev Ops: Build and manage automated CI/CD deployment pipelines using Azure Dev Ops across Development, QA, and Production environments.

- Platform Governance & Quality: Develop reusable data engineering frameworks,



ensure data quality, reliability, security, and governance across data platforms, and troubleshoot production pipeline failures and data issues.

- Engineering Standards & Leadership: Lead technical design discussions, architecture reviews, and code reviews while mentoring junior and mid-level data engineers.

Required Qualifications

- Experience: 8+ years of overall experience in Data Engineering, Data Development, Big Data, or related technology roles within enterprise environments.

- Databricks & PySpark: Strong hands-on experience with Azure Databricks, distributed computing, Spark architecture, and PySpark / Python.

- Advanced SQL: Advanced proficiency in SQL, including complex queries, data transformations, and performance tuning.

- Azure Data Ecosystem: Hands-on experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS / ADLS Gen2), and core Microsoft Azure cloud data services.

- Data Architecture & Dev Ops: Strong understanding of data warehousing, data modeling, ETL/ELT architecture, and practical experience implementing CI/CD using Azure Dev Ops across multi-stage environments (Dev, QA, Prod).

- Problem Solving: Proven analytical, troubleshooting, and problem-solving abilities within enterprise-scale, production data environments.

Preferred Qualifications

- Databricks Ecosystem: Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Lakehouse / Medallion Architecture (Bronze/Silver/Gold layers).





- Real-Time Data Streaming: Experience with Structured Streaming, Azure Event Hubs, or Apache Kafka.

- Azure Cloud Services: Familiarity with Azure Synapse Analytics, Azure SQL, and security management using Azure Key Vault.

- Infrastructure & Governance: Experience with Terraform / Infrastructure as Code, Git, data quality frameworks, monitoring/logging, and cloud cost optimization strategies.

Ideal Candidate Profile

The ideal candidate is a hands-on senior data engineer who can independently take ownership of enterprise data engineering projects across the end-to-end lifecycle:

$$text{Data Engineering} longrightarrow text{Databricks} longrightarrow text{PySpark} longrightarrow text{SQL} longrightarrow text{Azure} longrightarrow text{ADF} longrightarrow text{Data Architecture} longrightarrow text{CI/CD}$$

Candidates with proven experience designing scalable data platforms, optimizing Spark workloads, implementing production pipelines, and mentoring engineering teams will be strongly preferred.

About Our Client

Our client is a technology-driven organization developing and operating modern, cloud-based data platforms and enterprise data solutions. Working with large-scale data environments, they are investing heavily in modern Azure technologies to elevate data processing, analytics, reporting, and business intelligence capabilities through modern, collaborative development practices.

About Aura Recruitment Solutions

Aura Recruitment Solutions is a technology recruitment partner helping global organizations access high-quality technical talent from India. We specialize in recruiting pre-screened professionals across Data Engineering, AI & GenAI, Software Engineering, Cloud & Dev Ops, Cybersecurity, and Enterprise Technology.

📌 Senior Data Engineer – Azure Databricks (India)
🏢 Aura Recruitment Solutions
📍 India