Role Purpose :

As a Lead Data Engineer at Prevalent AI, you will lead the design and delivery of advanced data engineering projects for our clients. You will write code, review the team's code, and stay accountable for how the solution performs in production.

The work involves large data volumes, complex joins and transformations, and tight processing windows. Clients usually come to us when their pipelines are slow, expensive or difficult to change. Working out why, and then fixing it, is a large part of the role.

This role suits engineers who know Spark internals well enough to read the logs and explain what is going wrong, who ask the right questions about the data before they write code, and who can run a small team to a delivery date.

Key Accountabilities :
- Lead the design and delivery of client data engineering projects, from understanding the requirement through to production.
- Design pipelines that meet their SLA and continue to do so as data volumes grow.
- Diagnose failures and performance problems in large distributed jobs using logs and metrics.




- Set the data layout and storage strategy so jobs read only the data they need.
- Build reusable, configuration-driven components that cut development effort across the team.
- Review the team's code and agree the standards the team works to.
- Work with client teams to pin down the requirement and understand the data before development starts.
- Contribute to the platform work around delivery, including object storage, containerisation and CI/CD.
- Mentor developers, and document design decisions, tuning approaches and operational runbooks.

Skills & Experience :

Must Have :
- 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily.
- Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour.
- Experience

📌 Lead Data Engineer (Kochi)
🏢 Prevalent AI
📍 Kochi