Location: India

Location: India

Work Mode: Remote

Employment Type: Full-time

Experience: 5 to 10 Years

About the Role

We are seeking an experienced Senior Data Engineer Azure Databricks to design, develop, and optimize enterprise-scale data engineering solutions and modern cloud-based data platforms. The ideal candidate brings deep, hands-on expertise in Azure Databricks, PySpark, Python, SQL, Azure Data Factory (ADF), and Azure Cloud, alongside strong experience building scalable data pipelines and implementing automated CI/CD via Azure DevOps.

In this role, you will work closely with data architects, engineers, analysts, and business stakeholders to build reliable, high-performance data platforms, optimize distributed Spark workloads, and mentor growing engineering talent.

Key Responsibilities

- Data Pipeline Engineering: Design, build, and maintain scalable end-to-end ETL/ELT pipelines and complex processing workflows using Azure Databricks, PySpark, Python, and SQL.
- Orchestration & Integration: Build and orchestrate data ingestion workflows using Azure Data Factory (ADF) to integrate data from diverse structured and unstructured enterprise sources into Azure Data Lake Storage (ADLS Gen2).
- Lakehouse Architecture & Modeling: Implement scalable Lakehouse and Medallion architectures, designing data models and data warehousing solutions to support downstream analytics, reporting, and business applications.
- Performance & Cost Optimization: Audit and optimize Databricks jobs, Spark workloads, SQL queries, memory management, and data pipelines for peak performance and cloud cost efficiency.
- CI/CD & DevOps: Build and manage automated CI/CD deployment pipelines using Azure DevOps across Development, QA, and Production environments.
- Platform Governance & Quality: Develop reusable data engineering frameworks, ensure data quality, reliability, security, and governance across data platforms, and troubleshoot production pipeline failures and data issues.
- Engineering Standards & Leadership: Lead technical design discussions, architecture reviews, and code reviews while mentoring junior and mid-level data engineers.

Required Qualifications
- Experience: 8+ years of overall experience in Data Engineering, Data Development, Big Data, or related technology roles within enterprise environments.
- Databricks & PySpark: Strong hands-on experience with Azure Databricks, distributed computing, Spark architecture, and PySpark / Python.
- Advanced SQL: Advanced proficiency in SQL, including complex queries, data transformations, and performance tuning.
- Azure Data Ecosystem: Hands-on experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS / ADLS Gen2), and core Microsoft Azure cloud data services.
- Data Architecture & DevOps: Strong understanding of data warehousing, data modeling, ETL/ELT architecture, and practical experience implementing CI/CD using Azure DevOps across multi-stage environments (Dev, QA, Prod).
- Problem Solving: Proven analytical, troubleshooting, and problem-solving abilities within enterprise-scale, production data environments.

Preferred Qualifications




- Databricks Ecosystem: Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Lakehouse / Medallion Architecture (Bronze/Silver/Gold layers).
- Real-Time Data Streaming: Experience with Structured Streaming, Azure Event Hubs, or Apache Kafka.
- Azure Cloud Services: Familiarity with Azure Synapse Analytics, Azure SQL, and security management using Azure Key Vault.
- Infrastructure & Governance: Experience with Terraform / Infrastructure as Code, Git, data quality frameworks, monitoring/logging, and cloud cost optimization strategies.

Ideal Candidate Profile The ideal candidate is a hands-on senior data engineer who can independently take ownership of enterprise data engineering projects across the end-to-end lifecycle: $$text{Data Engineering} longrightarrow text{Databricks} longrightarrow text{PySpark} longrightarrow text{SQL} longrightarrow text{Azure} longrightarrow text{ADF} longrightarrow text{Data Architecture} longrightarrow text{CI/CD}$$

Candidates with proven experience designing scalable data platforms, optimizing Spark workloads, implementing production pipelines, and mentoring engineering teams will be strongly preferred.

About Our Client

Our client is a technology-driven organization developing and operating modern, cloud-based data platforms and enterprise data solutions. Working with large-scale data environments, they are investing heavily in modern Azure technologies to elevate data processing, analytics, reporting, and business intelligence capabilities through modern, collaborative development practices.

About Aura Recruitment Solutions

Aura Recruitment Solutions is a technology recruitment partner helping global organizations access high-quality technical talent from India. We specialize in recruiting pre-screened professionals across Data Engineering, AI & GenAI, Software Engineering, Cloud & DevOps, Cybersecurity, and Enterprise Technology.

Work Mode: Remote

Employment Type: Full time

Experience: 5+ Years

About the Role

We are seeking an experienced Senior DevOps / Site Reliability Engineer (SRE) to design, build, and maintain robust cloud infrastructure, automated CI/CD pipelines, and high-availability operational platforms. The ideal candidate brings strong hands-on expertise in AWS/Azure, Kubernetes, Terraform, Docker, CI/CD tools (GitHub Actions/Azure DevOps/Jenkins), and observability frameworks (Prometheus/Grafana/ELK).

In this role, you will work closely with software engineering, data, and security teams to drive platform reliability, infrastructure automation, developer efficiency, and system scalability.

Key Responsibilities

- Infrastructure as Code (IaC): Provision, manage, and scale cloud infrastructure using Terraform, CloudFormation,



or Ansible.
- Container Orchestration: Manage production-grade Kubernetes (EKS/AKS) clusters, helm charts, container runtimes, and ingress controllers.
- CI/CD Automation: Design, optimize, and secure automated build, test, and deployment pipelines using GitHub Actions, Azure DevOps, or Jenkins.
- Observability & Monitoring: Implement enterprise monitoring, log aggregation, and tracing systems using Prometheus, Grafana, Datadog, or ELK Stack.
- Reliability & Incident Management: Own system uptime, define SLOs/SLIs, maintain incident response protocols, and lead post-mortem root cause analyses (RCA).
- Security & Compliance: Enforce DevSecOps best practices, secret management (HashiCorp Vault/Key Vault), IAM policies, and cloud vulnerability scanning.
- Cost & Performance Optimization: Audit cloud resource utilization, implement autoscaling strategies, and optimize cloud spending.

Required Qualifications
- Experience: 6+ years of hands-on experience in DevOps, Site Reliability Engineering, or Cloud Infrastructure.
- Cloud Architecture: Deep expertise in AWS or Azure public cloud services and architecture.
- Containerization & Orchestration: Strong experience with Docker and production Kubernetes administration.
- Infrastructure as Code: Proficiency in Terraform for declarative infrastructure provisioning.
- CI/CD: Hands-on experience building multi-environment CI/CD pipelines with GitHub Actions, Azure DevOps, or Jenkins.
- Scripting: Robust scripting capabilities in Python, Bash, or Go for operational automation.
- Monitoring & Alerting: Solid background in setting up metrics, dashboards, and alerts via Prometheus and Grafana.

Preferred Qualifications
- Experience with Service Mesh (Istio/Linkerd), GitOps workflows (ArgoCD/Flux), and Serverless infrastructure.
- Familiarity with DevSecOps integrations (SonarQube, Trivy, Snyk).
- Certified Kubernetes Administrator (CKA) or AWS/Azure Solutions Architect/DevOps Certification.

About Our Client

About Our Client

Our client is an innovative financial technology platform headquartered in London, UK, delivering global payment rails, real-time transaction clearing, and open-banking API infrastructure. Operating under strict regulatory standards across European, APAC, and Americas markets, the organization prioritizes high availability, zero-downtime deployments, and robust security protocols.

Their engineering culture centers around a strong DevSecOps and Site Reliability Engineering (SRE) discipline, where automated chaos engineering, proactive observability, and self-healing infrastructure are core priorities. They foster a collaborative, blameless post-mortem environment where engineers have high ownership over system uptime, performance benchmarks, and modern multi-cloud deployment strategies.

About Aura Recruitment Solutions

Aura Recruitment Solutions helps global companies hire skilled technology professionals from India. We specialize in remote and offshore technology recruitment across Software Engineering, Data Engineering, Cloud & DevOps, AI, and Cybersecurity.

📌 Senior Data Engineer - Azure / Databricks (Bengaluru)
🏢 Aura Recruitment Solutions
📍 Bengaluru