Role & responsibilities

Lead PySpark Engineer (Distributed

Processing Platform)

Role Overview

We are seeking a highly experienced **Lead PySpark Engineer** to design, build, optimize, and lead

the development of large-scale distributed processing platform applications using Apache Spark.

This is a **hands-on technical leadership role** focused on architecting high-performance PySpark

solutions capable of processing **billions of records** with low latency and optimal resource

utilization.

This role is **not intended for ETL-only developers**. We are looking for engineers who deeply

understand Spark internals and can build highly optimized distributed processing systems at

enterprise scale.

Required Skills

- PySpark
- Python
- Apache Spark internals and distributed systems architecture
- Delta Lake , Parquet , Trino
- Linux , vi commands
- Workflow orchestration (Airflow)
- Minio
- Containerized deployment environments (Kubernetes)
- Git , Azure ADO
- Vscode
- Github Copilot

Preferred / Nice-to-Have

- Trade surveillance domain experience
- Financial services data engineering experience
- Time-series analytics
- Market data processing

Ideal Candidate Profile

The ideal candidate is a Spark architecture expert with proven real-world experience in:

- Distributed computing

- Memory and execution optimization
- Performance tuning at scale
- Framework development
- Scalable data platform engineering

You thrive in high-scale environments, combine deep technical execution with leadership, and can

deliver robust Spark systems for mission-critical workloads.

Key Responsibilities

1) Platform & Application Engineering

- Design and develop enterprise-scale distributed data processing applications using PySpark.
- Build reusable Spark frameworks and libraries that can be shared across multiple projects.
- Architect scalable batch and near real-time processing pipelines.
- Develop modular, configurable, and highly maintainable Spark applications.




- Establish coding standards and Spark engineering best practices across the team.

2) Performance Engineering (End-to-End Ownership)

Own Spark performance optimization across:

- Memory optimization
- CPU utilization optimization
- Shuffle reduction
- Stage optimization
- DAG optimization
- Data skew handling
- Job parallelism tuning
- Executor sizing
- Garbage Collection (GC) optimization

Be able to identify and resolve bottlenecks using:

- Spark UI
- Explain plans
- Event logs
- Executor/stage-level metrics

3) Distributed Processing Expertise

Demonstrate strong command of:

- Spark execution model

- Driver vs Executor architecture
- Lazy evaluation
- Catalyst optimizer
- Lineage and fault tolerance
- Adaptive Query Execution (AQE)

4) Data Partitioning & Data Layout

Hands-on expertise with:

- Partition strategy design
- Partition pruning
- Static partitioning
- Energetic partitioning

5) Join & Window Optimization

Extensive implementation and optimization experience for:

- Large-to-large dataset joins
- Window joins
- Time-series/time-based joins
- Interval joins

Must be able to select join strategies based on:

- Data volume
- Data skew
- Cluster behavior and resource profile

Strong window processing capabilities:

- Running aggregations
- Sliding windows
- Rolling windows
- Ranking
- Look-back calculations
- Time-based analytics
- Complex business-rule implementation

6) Spark Optimization Techniques

Applied experience with:

- Cache/Persist

- Checkpointing
- Compression
- Predicate pushdown
- Bucketing
- Z-ordering

7) Data Engineering Practices

- Delta Lake design and optimization
- Schema evolution and governance

8) Technical Leadership

- Lead architecture discussions for distributed Spark platforms.
- Conduct Spark code reviews and design reviews.
- Enforce engineering standards and quality practices.
- Mentor junior and senior engineers.
- Drive technical decisions across the data platform.
- Troubleshoot and lead resolution of critical production incidents.

📌 Tech Mahindra Hiring For Bigdata With Python and Pyspark (BLR/Chn) (Chennai)
🏢 Glauben Technologies
📍 Chennai