Title: Data Architect-Pyspark

Location: Bangalore

Exp: 8+ Yrs

:

Required Skills

PySpark and Python

Apache Spark internals and distributed systems architecture

Delta Lake, Parquet, and Trino

Linux and vi commands

Workflow orchestration using Airflow

MinIO

Containerized deployment environments using Kubernetes

Git and Azure DevOps (ADO)

Visual Studio Code (VS Code)

GitHub Copilot

Preferred / Nice-to-Have Skills

Trade surveillance domain experience

Financial services data engineering experience

Time-series analytics

Market data processing

Ideal Candidate Profile The ideal candidate is a Spark architecture expert with proven real-world experience in:

Distributed computing

Memory and execution optimization

Performance tuning at scale

Framework development

Scalable data platform engineering The candidate should thrive in high-scale environments, combine deep technical execution with leadership, and deliver robust Spark systems for mission-critical workloads.

Key Responsibilities

- Platform and Application Engineering

Design and develop enterprise-scale distributed data processing applications using PySpark.

Build reusable Spark frameworks and libraries that can be shared across multiple projects.

Architect scalable batch and near-real-time processing pipelines.

Develop modular, configurable, and highly maintainable Spark applications.

Establish coding standards and Spark engineering best practices across the team.

2.

Performance

Engineering

Own end-to-end Spark performance optimization, including:

Memory and CPU utilization optimization

Shuffle reduction

Stage and DAG optimization

Data skew handling

Job parallelism tuning

Executor sizing

Garbage Collection (GC) optimization





Identify and resolve performance bottlenecks using:

Spark UI

Explain plans

Event logs

Executor- and stage-level metrics

3.

Distributed Processing

Expertise

Demonstrate solid expertise in

Spark execution model

Driver and Executor architecture

Lazy evaluation

Catalyst Optimizer

Lineage and fault tolerance

Adaptive Query Execution (AQE)

- Data Partitioning and Data Layout

Design and implement effective partitioning strategies.

Optimize partition pruning.

Implement static and dynamic partitioning.

Improve data layout for efficient distributed processing.

- Join and Window Optimization

Implement and optimize large-to-large dataset joins.

Work with window joins, time-series joins, time-based joins, and interval joins.

Select appropriate join strategies based on data volume, data skew, cluster behavior, and resource profiles.

Implement advanced window processing, including:

Running aggregations

Sliding and rolling windows

Ranking

Look-back calculations

Time-based analytics

Complex business-rule implementation

6.

Spark Optimization

Techniques

Apply and optimize the following techniques:

Cache and Persist

Checkpointing

Compression

Predicate pushdown

Bucketing

Z-ordering

7.

Data Engineering

Practices

Design and optimize Delta Lake solutions.

Implement schema evolution and data governance practices.

8.

Technical

Leadership

Lead architecture discussions for distributed Spark platforms.

Conduct Spark code reviews and design reviews.

Establish and enforce engineering standards and quality practices.

Mentor junior and senior engineers.

Drive technical decisions across the data platform.

Troubleshoot critical production incidents and lead their resolution.

📌 Data Architect-Pyspark (Bengaluru)
🏢 NR Consulting
📍 Bengaluru