Senior Spark / PySpark Data Engineer
5+ Years Experience | Kolkata – Work from Office
Core Stack: Apache Spark • PySpark • Python • ETL/ELT • AWS • S3 • Glue • Redshift

Working Hours: Ability to work in the US Eastern Time Zone. Depending on project requirements, this may be adjusted to a half-day IST + half-day US EST schedule.

Immediate joiners preferred.

About the Job:
Web Spiders is looking for a Senior Spark / PySpark Data Engineer with solid hands-on experience in Apache Spark, PySpark, Python, and large-scale data engineering.

The ideal candidate will have strong experience designing and developing high-performance ETL/ELT pipelines and distributed data-processing solutions using Spark/PySpark, along with experience working with cloud-based data platforms and AWS services.

If Spark + PySpark + Python is your core expertise and you enjoy solving complex data-processing and scalability challenges, we'd love to hear from you.

What You'll Do:
Design, develop, and optimize large-scale ETL/ELT pipelines using Apache Spark and PySpark.




Develop scalable data transformation and processing solutions using PySpark and Python.
Build distributed data-processing applications capable of handling large volumes of data.
Develop reusable and maintainable Spark/PySpark frameworks and data-processing components.
Optimize Spark jobs for performance, scalability, memory utilization, and execution efficiency.
Work with complex transformations, joins, aggregations, partitioning, and large datasets.
Implement data validation, quality checks, error handling, and monitoring within data pipelines.
Work with AWS data services including EMR, Glue, S3, and Redshift.
Develop data pipelines supporting data lakes, warehouses, analytics, and downstream applications.
Troubleshoot production data pipeline and Spark processing issues.
Identify and resolve performance bottlenecks in Spark/PySpark workloads.
Collaborate with Data Engineering, Cloud, AI/ML, and Product teams to deliver reliab

📌 Senior Spark (Kolkata)
🏢 Web Spiders
📍 Kolkata