Job Description

Role Overview

n

We are looking for a Lead Data Engineer with solid experience building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure.

n

The role requires hands-on expertise in Python, PySpark, advanced SQL, Azure data services, Medallion Architecture, and deploying Apache Spark workloads on Kubernetes or Azure Kubernetes Service (AKS).

n

n

.Key Responsibilities

n

n
- Design and build batch, near-real-time, and streaming data pipelines on Azure.
n
- Develop Bronze, Silver, and Gold data layers using Medallion Architecture.
n
- Build and deploy containerized PySpark workloads on Kubernetes or AKS.
n
- Configure Spark drivers, executors, CPU, memory, scaling, dependencies, and storage access.
n
- Integrate data from REST APIs, SFTP, databases, files, enterprise systems, and Azure Event Hubs.
n
- Develop complex transformation, cleansing, enrichment, reconciliation, and validation workflows.
n
- Implement incremental loads, CDC, watermarking, deduplication, schema evolution, retries, and recovery.
n
- Optimize Spark jobs, partitioning, shuffles, joins, file sizes, and query performance.
n
- Implement monitoring, logging, alerting, audit controls, and data-quality checks.
n
- Build reusable Python, PySpark, and SQL components.
n
- Create CI/CD pipelines for Spark applications, Docker images, and Kubernetes deployments.
n




- Review technical designs and support data engineers with implementation standards.
n

n

n

Required Skills

n

n
- 6+ years of hands-on data engineering experience.
n
- Strong experience with Microsoft Azure data platforms.
n
- Advanced Python, PySpark, and SQL skills.
n
- Strong hands-on experience with: Apache Spark, Kubernetes and AKS, Docker, Azure Data Lake Storage Gen2, Azure Event Hubs, Azure DevOps and Git
n
- Experience deploying and operating Spark applications on Kubernetes.
n
- Strong understanding of Spark drivers, executors, resource allocation, partitioning, caching, broadcast joins, shuffle optimization, and skew handling.
n
- Experience with batch, streaming, ETL, ELT, and event-driven processing patterns.
n
- Experience implementing Medallion Architecture.
n
- Experience with REST APIs, SFTP, JSON, CSV, Parquet, Delta Lake, and relational databases.
n
- Experience with CDC, incremental processing, schema enforcement, and schema evolution.
n
- Strong understanding of data modelling, schema design, partitioning, and storage optimization.
n
- Experience with Kubernetes Jobs, Cron Jobs, Config Maps, Secrets, resource limits, node pools, and autoscaling.
n
- Experience implementing pipeline observability, data validation, monitoring, alerting, and error recovery.
n

n

📌 Lead Data Engineer (Pune)
🏢 ORMAE
📍 Pune