Position: Senior PySpark ETL Engineer

Role Summary
The Senior PySpark ETL Engineer is responsible for designing, building, optimizing, and operating scalable data pipelines using Apache Spark (PySpark). This role focuses on high volume batch (and optionally streaming) data processing, ensuring performance, reliability, data quality, and cost eciency across enterprise data platforms.

The position requires solid python, hands on Spark expertise, deep SQL and data modeling knowledge, and the ability to own pipelines end to end in production.

Mandatory Requirements

10 - 12 years of overall IT experience, with strong focus on data engineering and ETL.

3+ years of hands on experience with PySpark / Apache Spark in production environments.

Strong experience designing and implementing ETL / ELT pipelines at scale.
Excellent knowledge of SQL and relational data concepts.

Experience handling large datasets in distributed environments.

Strong ownership mindset, problem solving skills, and ability to independently handle production pipelines.

Core Technical Skills

PySpark & Spark Engineering





Deep expertise in PySpark:
DataFrames, Spark SQL, window functions, joins, aggregations
Spark execution model (DAGs, stages, tasks)

Strong hands on experience with:
Partitioning strategies
Shuffle optimization
Broadcast vs sort merge joins
Caching / persisting
Handling data skew and memory spills

Proven ability to debug and optimize slow Spark jobs.

ETL & Data Engineering:

Strong knowledge of ETL/ELT design patterns:
Incremental loads
Watermarking
Idempotent pipeline design
Reprocessing and backfill strategies

Experience implementing:
SCD Type 1 / Type 2
Deduplication and late arriving data handling
Ability to design reusable transformation frameworks and common utilities.
Experience building source to target reconciliation and data quality checks.

Data Storage & SQL
Excellent SQL skills including:
Complex joins
Subqueries and CTEs
Window functions
Query optimization

Expe

📌 Senior PySpark ETL Lead Engineer (Chennai)
🏢 Relevantz Technology Services
📍 Chennai