Web Spiders is looking for a Senior Spark / PySpark Data Engineer with strong hands-on experience in Apache Spark, PySpark, Python, and large-scale data engineering.

The ideal candidate will have solid experience designing and developing high-performance ETL/ELT pipelines and distributed data-processing solutions using Spark/PySpark, along with experience working with cloud-based data platforms and AWS services.

If Spark + PySpark + Python is your core expertise and you enjoy solving complex data-processing and scalability challenges, we'd love to hear from you.

5+ Years Experience | Kolkata – Work from Office

Core Stack: Apache Spark

PySpark
Python
ETL/ELT
AWS
S3
Glue
Redshift
Immediate joiners preferred.*

Working Hours: Ability to work in the US Eastern Time Zone. Depending on project requirements, this may be adjusted to a half-day IST + half-day US EST schedule.

What You'll Do

Design, develop, and optimize large-scale ETL/ELT pipelines using Apache Spark and PySpark.
Develop scalable data transformation and processing solutions using PySpark and Python.




Build distributed data-processing applications capable of handling large volumes of data.
Develop reusable and maintainable Spark/PySpark frameworks and data-processing components.
Optimize Spark jobs for performance, scalability, memory utilization, and execution efficiency.
Work with complex transformations, joins, aggregations, partitioning, and large datasets.
Implement data validation, quality checks, error handling, and monitoring within data pipelines.
Work with AWS data services including EMR, Glue, S3, and Redshift.
Develop data pipelines supporting data lakes, warehouses, analytics, and downstream applications.
Troubleshoot production data pipeline and Spark processing issues.
Identify and resolve performance bottlenecks in Spark/PySpark workloads.
Collaborate with Data Engineering, Cloud, AI/ML, and Product teams to deliver reliable data solutions.

‍

Must-Have Skills

5+ years of hands-on ex

📌 Senior Spark (Kolkata)
🏢 Web Spiders
📍 Kolkata