This role is for one of the Weekday's clients
Min Experience: 3+ years

Location: Bengaluru

JobType: full-time

We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems. The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.

This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.

Requirements
Key Responsibilities:
- Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala)
- Process and transform high-volume datasets (TBs of data) in distributed environments
- Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS)
- Work with object storage systems such as S3 and MinIO for efficient data access and storage




- Develop reusable and scalable data processing frameworks
- Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.)
- Manage and orchestrate workloads using HashiCorp Nomad
- Integrate data pipelines with PostgreSQL and other downstream systems
- Ensure data quality, consistency, and reliability across pipelines
- Troubleshoot production issues and perform root cause analysis
- Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.)

Required Skills:
- Strong programming experience in Scala
- Good working knowledge of Python (for auxiliary tasks, scripting, or ML integration)
- Deep expertise in Apache Spark (Core and SQL)
- Solid understanding of distributed data processing and large-scale systems
- Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters
- Hands-on experience with object s

📌 Big Data Developer (Bengaluru)
🏢 Weekday AI (YC W21)
📍 Bengaluru