This role is for one of the Weekday's clients
Min Experience: 3+ years

Location: Bengaluru

JobType: full-time

We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems. The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.

This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.

Requirements
Key Responsibilities:
Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala)
Process and transform high-volume datasets (TBs of data) in distributed environments
Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS)
Work with object storage systems such as S3 and MinIO for effective data access and storage




Develop reusable and scalable data processing frameworks
Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.)
Manage and orchestrate workloads using HashiCorp Nomad
Integrate data pipelines with PostgreSQL and other downstream systems
Ensure data quality, consistency, and reliability across pipelines
Troubleshoot production issues and perform root cause analysis
Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.)

Required Skills:
Strong programming experience in Scala
Positive working knowledge of Python (for auxiliary tasks, scripting, or ML integration)
Deep expertise in Apache Spark (Core and SQL)
Solid understanding of distributed data processing and large-scale systems
Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters
Hands-on experience with object s

📌 Big Data Developer Bengaluru
🏢 Weekday AI (YC W21)
📍 Bengaluru