This role is for one of Weekday’s clients
Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)

Min Experience: 8+ years
Location: Bengaluru, Chennai, Tamil Nadu, India
JobType: full time

We are looking for an experienced PySpark Developer with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands-on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision-making.

The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.

Requirements

Key Responsibilities

1. Data Pipeline Development
- Design, develop, and maintain scalable,



high-performance data pipelines using PySpark and Python.
- Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
- Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
- Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
- Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

2. PySpark Development and Optimization
- Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
- Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
- Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
- Work with large datasets while ensuring productive

📌 PySpark Developer (Chennai)
🏢 Weekday AI
📍 Chennai